public inbox for [email protected]help / color / mirror / Atom feed
[PATCH 1/8] cirrus: include hints how to install OS packages.. 12+ messages / 6 participants [nested] [flat]
* [PATCH 1/8] cirrus: include hints how to install OS packages.. @ 2022-01-17 06:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Justin Pryzby @ 2022-01-17 06:53 UTC (permalink / raw) This is useful for patches during development, but once a features is merged, new libraries should be added to the OS image files, rather than installed during every CI run forever into the future. --- .cirrus.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index dd96a97efe5..eda8ac9596c 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -73,10 +73,11 @@ task: chown -R postgres:postgres . mkdir -p ${CCACHE_DIR} chown -R postgres:postgres ${CCACHE_DIR} - setup_cores_script: | + setup_os_script: | mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kern.corefile='/tmp/cores/%N.%P.core' + #pkg install -y ... # NB: Intentionally build without --with-llvm. The freebsd image size is # already large enough to make VM startup slow, and even without llvm @@ -180,10 +181,12 @@ task: chown -R postgres:postgres ${CCACHE_DIR} echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf su postgres -c "ulimit -l -H && ulimit -l -S" - setup_cores_script: | + setup_os_script: | mkdir -m 770 /tmp/cores chown root:postgres /tmp/cores sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + #apt-get update + #apt-get -y install ... configure_script: | su postgres <<-EOF @@ -237,7 +240,7 @@ task: ulimit -a -H && ulimit -a -S export - setup_cores_script: + setup_os_script: - mkdir ${HOME}/cores - sudo sysctl kern.corefile="${HOME}/cores/core.%P" @@ -384,6 +387,9 @@ task: powershell -Command get-psdrive -psprovider filesystem set + setup_os_script: | + REM choco install -y ... + configure_script: # copy errors out when using forward slashes - copy src\tools\ci\windows_build_config.pl src\tools\msvc\config.pl -- 2.17.1 --4e5ZDkbgLEOfWmLx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-windows-add-compiler_warnings_script.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* RE: Popcount optimization using AVX512 @ 2024-03-19 22:56 Amonson, Paul D <[email protected]> 0 siblings, 2 replies; 12+ messages in thread From: Amonson, Paul D @ 2024-03-19 22:56 UTC (permalink / raw) To: Nathan Bossart <[email protected]>; David Rowley <[email protected]>; +Cc: Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> > -----Original Message----- > From: Nathan Bossart <[email protected]> > > Committed. Thanks for the suggestion and for reviewing! > > Paul, I suspect your patches will need to be rebased after commit cc4826d. > Would you mind doing so? Changed in this patch set. * Rebased. * Direct *slow* calls via macros as shown in example patch. * Changed the choose filename to be platform specific as suggested. * Falls back to intermediate "Fast" methods if AVX512 is not available at runtime. * inline used where is makes sense, remember using "extern" negates "inline". * Fixed comment issues pointed out in review. I tested building with and without TRY_POPCOUNT_FAST, for both configure and meson build systems, and ran in CI. Thanks, Paul Attachments: [application/octet-stream] v10-0001-Refactor-inlining-and-direct-calls-for-_slow-functio.patch (7.9K, ../../BL1PR11MB530414B036FD7E838830E398DC2C2@BL1PR11MB5304.namprd11.prod.outlook.com/2-v10-0001-Refactor-inlining-and-direct-calls-for-_slow-functio.patch) download | inline diff: From e9483d2354bd58210805b17fa78091ecc007c612 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 19 Mar 2024 13:35:53 -0700 Subject: [PATCH 1/3] [Refactor] inlining and direct calls for *_slow functions. Signed-off-by: Paul Amonson <[email protected]> --- src/backend/access/heap/visibilitymap.c | 6 +- src/include/nodes/bitmapset.h | 4 +- src/include/port/pg_bitutils.h | 17 +++-- src/port/pg_bitutils.c | 90 +++++-------------------- 4 files changed, 34 insertions(+), 83 deletions(-) diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c index 1ab6c865e3..28dc497b79 100644 --- a/src/backend/access/heap/visibilitymap.c +++ b/src/backend/access/heap/visibilitymap.c @@ -419,14 +419,14 @@ visibilitymap_count(Relation rel, BlockNumber *all_visible, BlockNumber *all_fro if (all_frozen == NULL) { for (i = 0; i < MAPSIZE / sizeof(uint64); i++) - nvisible += pg_popcount64(map[i] & VISIBLE_MASK64); + nvisible += PG_POPCOUNT64(map[i] & VISIBLE_MASK64); } else { for (i = 0; i < MAPSIZE / sizeof(uint64); i++) { - nvisible += pg_popcount64(map[i] & VISIBLE_MASK64); - nfrozen += pg_popcount64(map[i] & FROZEN_MASK64); + nvisible += PG_POPCOUNT64(map[i] & VISIBLE_MASK64); + nfrozen += PG_POPCOUNT64(map[i] & FROZEN_MASK64); } } diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h index 283bea5ea9..b5631d153e 100644 --- a/src/include/nodes/bitmapset.h +++ b/src/include/nodes/bitmapset.h @@ -77,11 +77,11 @@ typedef enum #if BITS_PER_BITMAPWORD == 32 #define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w) #define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w) -#define bmw_popcount(w) pg_popcount32(w) +#define bmw_popcount(w) PG_POPCOUNT32(w) #elif BITS_PER_BITMAPWORD == 64 #define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w) #define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w) -#define bmw_popcount(w) pg_popcount64(w) +#define bmw_popcount(w) PG_POPCOUNT64(w) #else #error "invalid BITS_PER_BITMAPWORD" #endif diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 53e5239717..477e00e0da 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -300,16 +300,23 @@ pg_ceil_log2_64(uint64 num) #ifdef TRY_POPCNT_FAST /* Attempt to use the POPCNT instruction, but perform a runtime check first */ +extern int pg_popcount32_slow(uint32 word); +extern int pg_popcount64_slow(uint64 word); +extern uint64 pg_popcount_slow(const char *buf, int bytes); extern PGDLLIMPORT int (*pg_popcount32) (uint32 word); extern PGDLLIMPORT int (*pg_popcount64) (uint64 word); extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); - +#define PG_POPCOUNT32(x) pg_popcount32(x) +#define PG_POPCOUNT64(x) pg_popcount64(x) +#define PG_POPCOUNT(x,y) pg_popcount(x,y) #else /* Use a portable implementation -- no need for a function pointer. */ -extern int pg_popcount32(uint32 word); -extern int pg_popcount64(uint64 word); -extern uint64 pg_popcount(const char *buf, int bytes); - +extern int pg_popcount32_slow(uint32 word); +extern int pg_popcount64_slow(uint64 word); +extern uint64 pg_popcount_slow(const char *buf, int bytes); +#define PG_POPCOUNT32(x) pg_popcount32_slow(x) +#define PG_POPCOUNT64(x) pg_popcount64_slow(x) +#define PG_POPCOUNT(x,y) pg_popcount_slow(x,y) #endif /* TRY_POPCNT_FAST */ /* diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index 1197696e97..e629969035 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -103,9 +103,9 @@ const uint8 pg_number_of_ones[256] = { 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8 }; -static inline int pg_popcount32_slow(uint32 word); -static inline int pg_popcount64_slow(uint64 word); -static uint64 pg_popcount_slow(const char *buf, int bytes); +// static inline int pg_popcount32_slow(uint32 word); +// static inline int pg_popcount64_slow(uint64 word); +// static uint64 pg_popcount_slow(const char *buf, int bytes); #ifdef TRY_POPCNT_FAST static bool pg_popcount_available(void); @@ -119,9 +119,6 @@ static uint64 pg_popcount_fast(const char *buf, int bytes); int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; -#endif /* TRY_POPCNT_FAST */ - -#ifdef TRY_POPCNT_FAST /* * Return true if CPUID indicates that the POPCNT instruction is available. @@ -148,8 +145,7 @@ pg_popcount_available(void) * the function pointers so that subsequent calls are routed directly to * the chosen implementation. */ -static int -pg_popcount32_choose(uint32 word) +static inline void set_function_pointers() { if (pg_popcount_available()) { @@ -163,45 +159,26 @@ pg_popcount32_choose(uint32 word) pg_popcount64 = pg_popcount64_slow; pg_popcount = pg_popcount_slow; } +} +static int +pg_popcount32_choose(uint32 word) +{ + set_function_pointers(); return pg_popcount32(word); } static int pg_popcount64_choose(uint64 word) { - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - + set_function_pointers(); return pg_popcount64(word); } static uint64 pg_popcount_choose(const char *buf, int bytes) { - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - + set_function_pointers(); return pg_popcount(buf, bytes); } @@ -243,7 +220,7 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_fast * Returns the number of 1-bits in buf */ -static uint64 +static inline uint64 pg_popcount_fast(const char *buf, int bytes) { uint64 popcnt = 0; @@ -256,7 +233,7 @@ pg_popcount_fast(const char *buf, int bytes) while (bytes >= 8) { - popcnt += pg_popcount64_fast(*words++); + popcnt += PG_POPCOUNT64(*words++); bytes -= 8; } @@ -270,7 +247,7 @@ pg_popcount_fast(const char *buf, int bytes) while (bytes >= 4) { - popcnt += pg_popcount32_fast(*words++); + popcnt += PG_POPCOUNT32(*words++); bytes -= 4; } @@ -292,7 +269,7 @@ pg_popcount_fast(const char *buf, int bytes) * pg_popcount32_slow * Return the number of 1 bits set in word */ -static inline int +inline int pg_popcount32_slow(uint32 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -314,7 +291,7 @@ pg_popcount32_slow(uint32 word) * pg_popcount64_slow * Return the number of 1 bits set in word */ -static inline int +inline int pg_popcount64_slow(uint64 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -342,7 +319,7 @@ pg_popcount64_slow(uint64 word) * pg_popcount_slow * Returns the number of 1-bits in buf */ -static uint64 +uint64 pg_popcount_slow(const char *buf, int bytes) { uint64 popcnt = 0; @@ -383,36 +360,3 @@ pg_popcount_slow(const char *buf, int bytes) return popcnt; } - -#ifndef TRY_POPCNT_FAST - -/* - * When the POPCNT instruction is not available, there's no point in using - * function pointers to vary the implementation between the fast and slow - * method. We instead just make these actual external functions when - * TRY_POPCNT_FAST is not defined. The compiler should be able to inline - * the slow versions here. - */ -int -pg_popcount32(uint32 word) -{ - return pg_popcount32_slow(word); -} - -int -pg_popcount64(uint64 word) -{ - return pg_popcount64_slow(word); -} - -/* - * pg_popcount - * Returns the number of 1-bits in buf - */ -uint64 -pg_popcount(const char *buf, int bytes) -{ - return pg_popcount_slow(buf, bytes); -} - -#endif /* !TRY_POPCNT_FAST */ -- 2.34.1 [application/octet-stream] v10-0002-Refactor-Seperated-slow-fast-and-choose-functionalit.patch (15.8K, ../../BL1PR11MB530414B036FD7E838830E398DC2C2@BL1PR11MB5304.namprd11.prod.outlook.com/3-v10-0002-Refactor-Seperated-slow-fast-and-choose-functionalit.patch) download | inline diff: From e5e4cac323b913e9fcdbd17d6b07316a21f7ff5c Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 19 Mar 2024 13:37:31 -0700 Subject: [PATCH 2/3] [Refactor] Seperated slow, fast, and choose functionality into files. Signed-off-by: Paul Amonson <[email protected]> --- contrib/intarray/_intbig_gist.c | 2 +- contrib/ltree/_ltree_gist.c | 2 +- contrib/pageinspect/heapfuncs.c | 4 +- contrib/pg_trgm/trgm_gist.c | 2 +- contrib/pg_walinspect/pg_walinspect.c | 2 +- src/backend/lib/bloomfilter.c | 2 +- src/backend/postmaster/syslogger.c | 2 +- src/backend/utils/adt/tsgistidx.c | 2 +- src/backend/utils/adt/varbit.c | 2 +- src/backend/utils/adt/varlena.c | 2 +- src/port/Makefile | 2 + src/port/meson.build | 2 + src/port/pg_bitutils.c | 171 +------------------------- src/port/pg_popcount_x86_64_accel.c | 101 +++++++++++++++ src/port/pg_popcount_x86_64_choose.c | 98 +++++++++++++++ 15 files changed, 215 insertions(+), 181 deletions(-) create mode 100644 src/port/pg_popcount_x86_64_accel.c create mode 100644 src/port/pg_popcount_x86_64_choose.c diff --git a/contrib/intarray/_intbig_gist.c b/contrib/intarray/_intbig_gist.c index 9699fbf3b4..a12ea7ed9b 100644 --- a/contrib/intarray/_intbig_gist.c +++ b/contrib/intarray/_intbig_gist.c @@ -210,7 +210,7 @@ g_intbig_compress(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount(sign, siglen); + return PG_POPCOUNT(sign, siglen); } static int diff --git a/contrib/ltree/_ltree_gist.c b/contrib/ltree/_ltree_gist.c index e89a39a5b5..bd66ec2e65 100644 --- a/contrib/ltree/_ltree_gist.c +++ b/contrib/ltree/_ltree_gist.c @@ -180,7 +180,7 @@ _ltree_union(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount((const char *) sign, siglen); + return PG_POPCOUNT((const char *) sign, siglen); } static int diff --git a/contrib/pageinspect/heapfuncs.c b/contrib/pageinspect/heapfuncs.c index 3faeabc711..089842962f 100644 --- a/contrib/pageinspect/heapfuncs.c +++ b/contrib/pageinspect/heapfuncs.c @@ -527,8 +527,8 @@ heap_tuple_infomask_flags(PG_FUNCTION_ARGS) if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) elog(ERROR, "return type must be a row type"); - bitcnt = pg_popcount((const char *) &t_infomask, sizeof(uint16)) + - pg_popcount((const char *) &t_infomask2, sizeof(uint16)); + bitcnt = PG_POPCOUNT((const char *) &t_infomask, sizeof(uint16)) + + PG_POPCOUNT((const char *) &t_infomask2, sizeof(uint16)); /* If no flags, return a set of empty arrays */ if (bitcnt <= 0) diff --git a/contrib/pg_trgm/trgm_gist.c b/contrib/pg_trgm/trgm_gist.c index 9ef2e38560..850316196f 100644 --- a/contrib/pg_trgm/trgm_gist.c +++ b/contrib/pg_trgm/trgm_gist.c @@ -648,7 +648,7 @@ gtrgm_same(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount(sign, siglen); + return PG_POPCOUNT(sign, siglen); } static int diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index ee2918726d..93a7b4842a 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -303,7 +303,7 @@ GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, block_fpi_len = blk->bimg_len; /* Construct and save block_fpi_info */ - bitcnt = pg_popcount((const char *) &blk->bimg_info, + bitcnt = PG_POPCOUNT((const char *) &blk->bimg_info, sizeof(uint8)); flags = (Datum *) palloc0(sizeof(Datum) * bitcnt); if ((blk->bimg_info & BKPIMAGE_HAS_HOLE) != 0) diff --git a/src/backend/lib/bloomfilter.c b/src/backend/lib/bloomfilter.c index 360d21ca45..c01b069c01 100644 --- a/src/backend/lib/bloomfilter.c +++ b/src/backend/lib/bloomfilter.c @@ -187,7 +187,7 @@ double bloom_prop_bits_set(bloom_filter *filter) { int bitset_bytes = filter->m / BITS_PER_BYTE; - uint64 bits_set = pg_popcount((char *) filter->bitset, bitset_bytes); + uint64 bits_set = PG_POPCOUNT((char *) filter->bitset, bitset_bytes); return bits_set / (double) filter->m; } diff --git a/src/backend/postmaster/syslogger.c b/src/backend/postmaster/syslogger.c index 08efe74cc9..85c57b3154 100644 --- a/src/backend/postmaster/syslogger.c +++ b/src/backend/postmaster/syslogger.c @@ -898,7 +898,7 @@ process_pipe_input(char *logbuffer, int *bytes_in_logbuffer) if (p.nuls[0] == '\0' && p.nuls[1] == '\0' && p.len > 0 && p.len <= PIPE_MAX_PAYLOAD && p.pid != 0 && - pg_popcount((char *) &dest_flags, 1) == 1) + PG_POPCOUNT((char *) &dest_flags, 1) == 1) { List *buffer_list; ListCell *cell; diff --git a/src/backend/utils/adt/tsgistidx.c b/src/backend/utils/adt/tsgistidx.c index 5698ee5502..d7a76faf31 100644 --- a/src/backend/utils/adt/tsgistidx.c +++ b/src/backend/utils/adt/tsgistidx.c @@ -489,7 +489,7 @@ gtsvector_same(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount(sign, siglen); + return PG_POPCOUNT(sign, siglen); } static int diff --git a/src/backend/utils/adt/varbit.c b/src/backend/utils/adt/varbit.c index 8fcf3fb731..3f287cd54d 100644 --- a/src/backend/utils/adt/varbit.c +++ b/src/backend/utils/adt/varbit.c @@ -1212,7 +1212,7 @@ bit_bit_count(PG_FUNCTION_ARGS) { VarBit *arg = PG_GETARG_VARBIT_P(0); - PG_RETURN_INT64(pg_popcount((char *) VARBITS(arg), VARBITBYTES(arg))); + PG_RETURN_INT64(PG_POPCOUNT((char *) VARBITS(arg), VARBITBYTES(arg))); } /* diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 8d28dd42ce..809e6a59ab 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -3151,7 +3151,7 @@ bytea_bit_count(PG_FUNCTION_ARGS) { bytea *t1 = PG_GETARG_BYTEA_PP(0); - PG_RETURN_INT64(pg_popcount(VARDATA_ANY(t1), VARSIZE_ANY_EXHDR(t1))); + PG_RETURN_INT64(PG_POPCOUNT(VARDATA_ANY(t1), VARSIZE_ANY_EXHDR(t1))); } /* diff --git a/src/port/Makefile b/src/port/Makefile index dcc8737e68..1499985dfc 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -44,6 +44,8 @@ OBJS = \ noblock.o \ path.o \ pg_bitutils.o \ + pg_popcount_x86_64_choose.o \ + pg_popcount_x86_64_accel.o \ pg_strong_random.o \ pgcheckdir.o \ pgmkdirp.o \ diff --git a/src/port/meson.build b/src/port/meson.build index 92b593e6ef..cf6e9fa06c 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -7,6 +7,8 @@ pgport_sources = [ 'noblock.c', 'path.c', 'pg_bitutils.c', + 'pg_popcount_x86_64_choose.c', + 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index e629969035..f08820b35b 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -11,14 +11,6 @@ *------------------------------------------------------------------------- */ #include "c.h" - -#ifdef HAVE__GET_CPUID -#include <cpuid.h> -#endif -#ifdef HAVE__CPUID -#include <intrin.h> -#endif - #include "port/pg_bitutils.h" @@ -103,167 +95,6 @@ const uint8 pg_number_of_ones[256] = { 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8 }; -// static inline int pg_popcount32_slow(uint32 word); -// static inline int pg_popcount64_slow(uint64 word); -// static uint64 pg_popcount_slow(const char *buf, int bytes); - -#ifdef TRY_POPCNT_FAST -static bool pg_popcount_available(void); -static int pg_popcount32_choose(uint32 word); -static int pg_popcount64_choose(uint64 word); -static uint64 pg_popcount_choose(const char *buf, int bytes); -static inline int pg_popcount32_fast(uint32 word); -static inline int pg_popcount64_fast(uint64 word); -static uint64 pg_popcount_fast(const char *buf, int bytes); - -int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; -int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; -uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; - -/* - * Return true if CPUID indicates that the POPCNT instruction is available. - */ -static bool -pg_popcount_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 23)) != 0; /* POPCNT */ -} - -/* - * These functions get called on the first call to pg_popcount32 etc. - * They detect whether we can use the asm implementations, and replace - * the function pointers so that subsequent calls are routed directly to - * the chosen implementation. - */ -static inline void set_function_pointers() -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } -} - -static int -pg_popcount32_choose(uint32 word) -{ - set_function_pointers(); - return pg_popcount32(word); -} - -static int -pg_popcount64_choose(uint64 word) -{ - set_function_pointers(); - return pg_popcount64(word); -} - -static uint64 -pg_popcount_choose(const char *buf, int bytes) -{ - set_function_pointers(); - return pg_popcount(buf, bytes); -} - -/* - * pg_popcount32_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount32_fast(uint32 word) -{ -#ifdef _MSC_VER - return __popcnt(word); -#else - uint32 res; - -__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif -} - -/* - * pg_popcount64_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount64_fast(uint64 word) -{ -#ifdef _MSC_VER - return __popcnt64(word); -#else - uint64 res; - -__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif -} - -/* - * pg_popcount_fast - * Returns the number of 1-bits in buf - */ -static inline uint64 -pg_popcount_fast(const char *buf, int bytes) -{ - uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) - { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += PG_POPCOUNT64(*words++); - bytes -= 8; - } - - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += PG_POPCOUNT32(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif - - /* Process any remaining bytes */ - while (bytes--) - popcnt += pg_number_of_ones[(unsigned char) *buf++]; - - return popcnt; -} - -#endif /* TRY_POPCNT_FAST */ - /* * pg_popcount32_slow @@ -319,7 +150,7 @@ pg_popcount64_slow(uint64 word) * pg_popcount_slow * Returns the number of 1-bits in buf */ -uint64 +inline uint64 pg_popcount_slow(const char *buf, int bytes) { uint64 popcnt = 0; diff --git a/src/port/pg_popcount_x86_64_accel.c b/src/port/pg_popcount_x86_64_accel.c new file mode 100644 index 0000000000..d63e8aa30f --- /dev/null +++ b/src/port/pg_popcount_x86_64_accel.c @@ -0,0 +1,101 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_accel.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_accel.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" +#include "port/pg_bitutils.h" + +#ifdef TRY_POPCNT_FAST +int pg_popcount32_fast(uint32 word); +int pg_popcount64_fast(uint64 word); +uint64 pg_popcount_fast(const char *buf, int bytes); + +/* + * pg_popcount32_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount32_fast(uint32 word) +{ +#ifdef _MSC_VER + return __popcnt(word); +#else + uint32 res; + +__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount64_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount64_fast(uint64 word) +{ +#ifdef _MSC_VER + return __popcnt64(word); +#else + uint64 res; + +__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount_fast + * Returns the number of 1-bits in buf + */ +uint64 +pg_popcount_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + +#if SIZEOF_VOID_P >= 8 + /* Process in 64-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(8, buf)) + { + const uint64 *words = (const uint64 *) buf; + + while (bytes >= 8) + { + popcnt += PG_POPCOUNT64(*words++); + bytes -= 8; + } + + buf = (const char *) words; + } +#else + /* Process in 32-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(4, buf)) + { + const uint32 *words = (const uint32 *) buf; + + while (bytes >= 4) + { + popcnt += PG_POPCOUNT32(*words++); + bytes -= 4; + } + + buf = (const char *) words; + } +#endif + + /* Process any remaining bytes */ + while (bytes--) + popcnt += pg_number_of_ones[(unsigned char) *buf++]; + + return popcnt; +} + +#endif /* TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_choose.c b/src/port/pg_popcount_x86_64_choose.c new file mode 100644 index 0000000000..1a0022a0b3 --- /dev/null +++ b/src/port/pg_popcount_x86_64_choose.c @@ -0,0 +1,98 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_choose.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_choose.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" + +#ifdef HAVE__GET_CPUID +#include <cpuid.h> +#endif +#ifdef HAVE__CPUID +#include <intrin.h> +#endif + +#include "port/pg_bitutils.h" + +#ifdef TRY_POPCNT_FAST +int pg_popcount32_fast(uint32 word); +int pg_popcount64_fast(uint64 word); +uint64 pg_popcount_fast(const char *buf, int bytes); + +static int pg_popcount32_choose(uint32 word); +static int pg_popcount64_choose(uint64 word); +static uint64 pg_popcount_choose(const char *buf, int bytes); + +int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; +int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; +uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; + +/* + * Return true if CPUID indicates that the POPCNT instruction is available. + */ +static bool +pg_popcount_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + +#if defined(HAVE__GET_CPUID) + __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUID) + __cpuid(exx, 1); +#else +#error cpuid instruction not available +#endif + + return (exx[2] & (1 << 23)) != 0; /* POPCNT */ +} + +/* + * These functions get called on the first call to pg_popcount32 etc. + * They detect whether we can use the asm implementations, and replace + * the function pointers so that subsequent calls are routed directly to + * the chosen implementation. + */ +static inline void set_function_pointers() +{ + if (pg_popcount_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + pg_popcount32 = pg_popcount32_slow; + pg_popcount64 = pg_popcount64_slow; + pg_popcount = pg_popcount_slow; + } +} + +static int +pg_popcount32_choose(uint32 word) +{ + set_function_pointers(); + return pg_popcount32(word); +} + +static int +pg_popcount64_choose(uint64 word) +{ + set_function_pointers(); + return pg_popcount64(word); +} + +static uint64 +pg_popcount_choose(const char *buf, int bytes) +{ + set_function_pointers(); + return pg_popcount(buf, bytes); +} +#endif /* TRY_POPCNT_FAST */ -- 2.34.1 [application/octet-stream] v10-0003-Feature-Add-POPCNT512-accelerated-functionality-for-.patch (23.5K, ../../BL1PR11MB530414B036FD7E838830E398DC2C2@BL1PR11MB5304.namprd11.prod.outlook.com/4-v10-0003-Feature-Add-POPCNT512-accelerated-functionality-for-.patch) download | inline diff: From b586e558d8eec0f5d5c45dfabec317ad5300e8d6 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 19 Mar 2024 13:40:04 -0700 Subject: [PATCH 3/3] [Feature] Add POPCNT512 accelerated functionality for x86_64. Signed-off-by: Paul Amonson <[email protected]> --- config/c-compiler.m4 | 37 +++++ configure | 205 +++++++++++++++++++++++++++ configure.ac | 44 ++++++ meson.build | 72 ++++++++++ src/Makefile.global.in | 1 + src/backend/commands/user.c | 2 +- src/include/pg_config.h.in | 12 ++ src/include/port/pg_bitutils.h | 6 +- src/makefiles/meson.build | 1 + src/port/Makefile | 5 + src/port/meson.build | 6 +- src/port/pg_popcount_x86_64_accel.c | 33 +++++ src/port/pg_popcount_x86_64_choose.c | 68 ++++++++- 13 files changed, 479 insertions(+), 13 deletions(-) diff --git a/config/c-compiler.m4 b/config/c-compiler.m4 index 3268a780bb..94e3e713aa 100644 --- a/config/c-compiler.m4 +++ b/config/c-compiler.m4 @@ -694,3 +694,40 @@ if test x"$Ac_cachevar" = x"yes"; then fi undefine([Ac_cachevar])dnl ])# PGAC_LOONGARCH_CRC32C_INTRINSICS + +# PGAC_AVX512_POPCNT_INTRINSICS +# --------------------------- +# Check if the compiler supports the x86_64 AVX512 POPCNT instructions using +# intrinsics used in CPUID features AVX512F and AVX512VPOPCNTDQ. +# +# Optional compiler flags can be passed as argument (e.g. -mavx512vpopcntdq). +# If the intrinsics are supported then pgac_avx512_popcnt_intrinsics and +# CFLAGS_AVX512_POPCNT are set. +AC_DEFUN([PGAC_AVX512_POPCNT_INTRINSICS], +[define([Ac_cachevar], [AS_TR_SH([pgac_cv_avx512_popcnt_intrinsics_$1])])dnl +AC_CACHE_CHECK([for _mm512_popcnt_epi64 with CFLAGS=$1], [Ac_cachevar], +[pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS $1" +AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h>], + [const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0;])], + [Ac_cachevar=yes], + [Ac_cachevar=no]) +CFLAGS="$pgac_save_CFLAGS"]) +if test x"$Ac_cachevar" = x"yes"; then + CFLAGS_AVX512_POPCNT="$1" + pgac_avx512_popcnt_intrinsics=yes +fi +undefine([Ac_cachevar])dnl +])# PGAC_AVX512_POPCNT_INTRINSICS diff --git a/configure b/configure index 36feeafbb2..0fbfc7c78f 100755 --- a/configure +++ b/configure @@ -647,6 +647,7 @@ MSGFMT_FLAGS MSGFMT PG_CRC32C_OBJS CFLAGS_CRC +CFLAGS_AVX512_POPCNT LIBOBJS OPENSSL ZSTD @@ -17404,6 +17405,41 @@ $as_echo "#define HAVE__GET_CPUID 1" >>confdefs.h fi +# Check for x86 cpuid_count instruction +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __get_cpuid_count" >&5 +$as_echo_n "checking for __get_cpuid_count... " >&6; } +if ${pgac_cv__get_cpuid_count+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <cpuid.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__get_cpuid_count="yes" +else + pgac_cv__get_cpuid_count="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__get_cpuid_count" >&5 +$as_echo "$pgac_cv__get_cpuid_count" >&6; } +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + +$as_echo "#define HAVE__GET_CPUID_COUNT 1" >>confdefs.h + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuid" >&5 $as_echo_n "checking for __cpuid... " >&6; } if ${pgac_cv__cpuid+:} false; then : @@ -17438,6 +17474,175 @@ $as_echo "#define HAVE__CPUID 1" >>confdefs.h fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuidex" >&5 +$as_echo_n "checking for __cpuidex... " >&6; } +if ${pgac_cv__cpuidex+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <intrin.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__cpuidex="yes" +else + pgac_cv__cpuidex="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__cpuidex" >&5 +$as_echo "$pgac_cv__cpuidex" >&6; } +if test x"$pgac_cv__cpuidex" = x"yes"; then + +$as_echo "#define HAVE__CPUIDEX 1" >>confdefs.h + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __immintrin" >&5 +$as_echo_n "checking for __immintrin... " >&6; } +if ${pgac_cv__immintrin+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +int +main () +{ +/* Don't exclude code so added return. */ + return 1701; + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__immintrin="yes" +else + pgac_cv__immintrin="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__immintrin" >&5 +$as_echo "$pgac_cv__immintrin" >&6; } +if test x"$pgac_cv__immintrin" = x"yes"; then + +$as_echo "#define HAVE__IMMINTRIN 1" >>confdefs.h + +fi + +# Check for Intel AVX512 intrinsics to do POPCNT calculations. +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics_+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS " +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics_=yes +else + pgac_cv_avx512_popcnt_intrinsics_=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics_" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics_" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics_" = x"yes"; then + CFLAGS_AVX512_POPCNT="" + pgac_avx512_popcnt_intrinsics=yes +fi + +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS -mavx512vpopcntdq" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=yes +else + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" = x"yes"; then + CFLAGS_AVX512_POPCNT="-mavx512vpopcntdq" + pgac_avx512_popcnt_intrinsics=yes +fi + + +$as_echo "#define HAVE__AVX512_POPCNT 1" >>confdefs.h + +fi + + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/configure.ac b/configure.ac index 57f734879e..3c741d457d 100644 --- a/configure.ac +++ b/configure.ac @@ -2052,6 +2052,18 @@ if test x"$pgac_cv__get_cpuid" = x"yes"; then AC_DEFINE(HAVE__GET_CPUID, 1, [Define to 1 if you have __get_cpuid.]) fi +# Check for x86 cpuid_count instruction +AC_CACHE_CHECK([for __get_cpuid_count], [pgac_cv__get_cpuid_count], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <cpuid.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + ]])], + [pgac_cv__get_cpuid_count="yes"], + [pgac_cv__get_cpuid_count="no"])]) +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + AC_DEFINE(HAVE__GET_CPUID_COUNT, 1, [Define to 1 if you have __get_cpuid.]) +fi + AC_CACHE_CHECK([for __cpuid], [pgac_cv__cpuid], [AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], [[unsigned int exx[4] = {0, 0, 0, 0}; @@ -2063,6 +2075,38 @@ if test x"$pgac_cv__cpuid" = x"yes"; then AC_DEFINE(HAVE__CPUID, 1, [Define to 1 if you have __cpuid.]) fi +AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + ]])], + [pgac_cv__cpuidex="yes"], + [pgac_cv__cpuidex="no"])]) +if test x"$pgac_cv__cpuidex" = x"yes"; then + AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.]) +fi + +AC_CACHE_CHECK([for __immintrin], [pgac_cv__immintrin], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h>], + [[/* Don't exclude code so added return. */ + return 1701; + ]])], + [pgac_cv__immintrin="yes"], + [pgac_cv__immintrin="no"])]) +if test x"$pgac_cv__immintrin" = x"yes"; then + AC_DEFINE(HAVE__IMMINTRIN, 1, [Define to 1 if you have immintrin.]) +fi + +# Check for Intel AVX512 intrinsics to do POPCNT calculations. +# +PGAC_AVX512_POPCNT_INTRINSICS([]) +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + PGAC_AVX512_POPCNT_INTRINSICS([-mavx512vpopcntdq]) + AC_DEFINE(HAVE__AVX512_POPCNT, 1, [Define to 1 if you have cpu + support for AVX512 POPCNT.]) +fi +AC_SUBST(CFLAGS_AVX512_POPCNT) + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/meson.build b/meson.build index c8fdfeb0ec..d661405fee 100644 --- a/meson.build +++ b/meson.build @@ -1783,6 +1783,37 @@ elif cc.links(''' endif +if cc.links(''' + #include <cpuid.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + } + ''', name: '__get_cpuid_count', + args: test_c_args) + cdata.set('HAVE__GET_CPUID_COUNT', 1) +elif cc.links(''' + #include <intrin.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __cpuidex(exx, 7, 0); + } + ''', name: '__cpuidex', + args: test_c_args) + cdata.set('HAVE__CPUIDEX', 1) +endif + + +# Check for header immintrin.h +if cc.has_header('immintrin.h', + include_directories: postgres_inc, args: test_c_args) + cdata.set('HAVE__IMMINTRIN', 1, + description: 'Define to 1 if you have the immintrin.h header file.') +endif + + # Defend against clang being used on x86-32 without SSE2 enabled. As current # versions of clang do not understand -fexcess-precision=standard, the use of # x87 floating point operations leads to problems like isinf possibly returning @@ -2157,6 +2188,47 @@ elif host_cpu == 'ppc' or host_cpu == 'ppc64' endif +############################################################### +# AVX 512 POPCNT Intrinsic check +############################################################### +have_avx512_popcnt = false +cflags_avx512_popcnt = [] +if host_cpu == 'x86_64' + test_flags = ['-mavx512vpopcntdq'] + if host_system == 'windows' + test_flags = ['/arch:AVX512'] + endif + prog = ''' + #include <immintrin.h> + #include <stdint.h> + #include <stdlib.h> + #include <string.h> + void main(void) + { + const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + }''' + if cc.links(prog, name: '_mm512_* methods with -mavx512vpopcntdq flag.', + args: test_c_args + test_flags) + have_avx512_popcnt = true + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cdata.set('HAVE__AVX512_POPCNT', 1) + cflags_avx512_popcnt = test_flags + else + have_avx512_popcnt = false + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cflags_avx512_popcnt = [] + endif # compile/link test +endif # host_cpu check + ############################################################### # Library / OS tests diff --git a/src/Makefile.global.in b/src/Makefile.global.in index 8b3f8c24e0..089f49b7f3 100644 --- a/src/Makefile.global.in +++ b/src/Makefile.global.in @@ -263,6 +263,7 @@ CXXFLAGS_SL_MODULE = @CXXFLAGS_SL_MODULE@ CFLAGS_UNROLL_LOOPS = @CFLAGS_UNROLL_LOOPS@ CFLAGS_VECTORIZE = @CFLAGS_VECTORIZE@ CFLAGS_CRC = @CFLAGS_CRC@ +CFLAGS_AVX512_POPCNT = @CFLAGS_AVX512_POPCNT@ PERMIT_DECLARATION_AFTER_STATEMENT = @PERMIT_DECLARATION_AFTER_STATEMENT@ CXXFLAGS = @CXXFLAGS@ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index c75cde2e8e..77d72daa87 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -2330,7 +2330,7 @@ plan_single_revoke(CatCList *memlist, RevokeRoleGrantAction *actions, * wouldn't work properly if such syntax were added, so assert that our * caller isn't trying to do that. */ - Assert(pg_popcount32(popt->specified) <= 1); + Assert(PG_POPCOUNT32(popt->specified) <= 1); for (i = 0; i < memlist->n_members; ++i) { diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 591e1ca3df..33a831e768 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -558,6 +558,18 @@ /* Define to 1 if you have __get_cpuid. */ #undef HAVE__GET_CPUID +/* Define to 1 if you have __get_cpuid_count. */ +#undef HAVE__GET_CPUID_COUNT + +/* Define to 1 if you have __get_cpuidex. */ +#undef HAVE__GET_CPUIDEX + +/* Define to 1 if you have immintrin. */ +#undef HAVE__IMMINTRIN + +/* Define to 1 if you have AVX512. */ +#undef HAVE__AVX512_POPCNT + /* Define to 1 if your compiler understands _Static_assert. */ #undef HAVE__STATIC_ASSERT diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 477e00e0da..ead4f79def 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -300,9 +300,9 @@ pg_ceil_log2_64(uint64 num) #ifdef TRY_POPCNT_FAST /* Attempt to use the POPCNT instruction, but perform a runtime check first */ -extern int pg_popcount32_slow(uint32 word); -extern int pg_popcount64_slow(uint64 word); -extern uint64 pg_popcount_slow(const char *buf, int bytes); +extern inline int pg_popcount32_slow(uint32 word); +extern inline int pg_popcount64_slow(uint64 word); +extern inline uint64 pg_popcount_slow(const char *buf, int bytes); extern PGDLLIMPORT int (*pg_popcount32) (uint32 word); extern PGDLLIMPORT int (*pg_popcount64) (uint64 word); extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build index b0f4178b3d..ee3647282e 100644 --- a/src/makefiles/meson.build +++ b/src/makefiles/meson.build @@ -100,6 +100,7 @@ pgxs_kv = { ' '.join(cflags_no_decl_after_statement), 'CFLAGS_CRC': ' '.join(cflags_crc), + 'CFLAGS_AVX512_POPCNT': ' '.join(cflags_avx512_popcnt), 'CFLAGS_UNROLL_LOOPS': ' '.join(unroll_loops_cflags), 'CFLAGS_VECTORIZE': ' '.join(vectorize_cflags), diff --git a/src/port/Makefile b/src/port/Makefile index 1499985dfc..66ef151565 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -89,6 +89,11 @@ pg_crc32c_sse42.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_shlib.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_srv.o: CFLAGS+=$(CFLAGS_CRC) +# Newer Intel processors can use AVX-512 POPCNT Capabilities (01/30/2024) +pg_popcount_x86_64_accel.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_shlib.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_srv.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) + # all versions of pg_crc32c_armv8.o need CFLAGS_CRC pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC) diff --git a/src/port/meson.build b/src/port/meson.build index cf6e9fa06c..0647e7a4f7 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,7 +8,6 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_x86_64_choose.c', - 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', @@ -86,6 +85,7 @@ replace_funcs_pos = [ ['pg_crc32c_sse42', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK', 'crc'], ['pg_crc32c_sse42_choose', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], ['pg_crc32c_sb8', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], + ['pg_popcount_x86_64_accel', 'USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 'avx512'], # arm / aarch64 ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C'], @@ -100,8 +100,8 @@ replace_funcs_pos = [ ['pg_crc32c_sb8', 'USE_SLICING_BY_8_CRC32C'], ] -pgport_cflags = {'crc': cflags_crc} -pgport_sources_cflags = {'crc': []} +pgport_cflags = {'crc': cflags_crc, 'avx512': cflags_avx512_popcnt} +pgport_sources_cflags = {'crc': [], 'avx512': []} foreach f : replace_funcs_neg func = f.get(0) diff --git a/src/port/pg_popcount_x86_64_accel.c b/src/port/pg_popcount_x86_64_accel.c index d63e8aa30f..f65184ddce 100644 --- a/src/port/pg_popcount_x86_64_accel.c +++ b/src/port/pg_popcount_x86_64_accel.c @@ -13,10 +13,15 @@ #include "c.h" #include "port/pg_bitutils.h" +#if defined(HAVE__IMMINTRIN) +#include <immintrin.h> +#endif + #ifdef TRY_POPCNT_FAST int pg_popcount32_fast(uint32 word); int pg_popcount64_fast(uint64 word); uint64 pg_popcount_fast(const char *buf, int bytes); +uint64 pg_popcount512_fast(const char *buf, int bytes); /* * pg_popcount32_fast @@ -98,4 +103,32 @@ pg_popcount_fast(const char *buf, int bytes) return popcnt; } +/* + * Use AVX-512 Intrinsics for supported CPUs or fall back the non-152 fast + * implem entation and use the best 64 bit fast methods. If no fast + * methods are used this will fall back to __builtin_* or pure software. + */ +uint64 +pg_popcount512_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + #if defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 + __m512i accumulator = _mm512_setzero_si512(); + + while (bytes >= 64) + { + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + + accumulator = _mm512_add_epi64(accumulator, p); + bytes -= 64; + buf += 64; + } + + popcnt = _mm512_reduce_add_epi64(accumulator); +#endif /* defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 */ + + /* Process any remaining bytes */ + return popcnt + pg_popcount_fast(buf, bytes); +} #endif /* TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_choose.c b/src/port/pg_popcount_x86_64_choose.c index 1a0022a0b3..d25d251e9b 100644 --- a/src/port/pg_popcount_x86_64_choose.c +++ b/src/port/pg_popcount_x86_64_choose.c @@ -25,6 +25,7 @@ int pg_popcount32_fast(uint32 word); int pg_popcount64_fast(uint64 word); uint64 pg_popcount_fast(const char *buf, int bytes); +uint64 pg_popcount512_fast(const char *buf, int bytes); static int pg_popcount32_choose(uint32 word); static int pg_popcount64_choose(uint64 word); @@ -53,6 +54,52 @@ pg_popcount_available(void) return (exx[2] & (1 << 23)) != 0; /* POPCNT */ } +/* + * Return true if CPUID indicates that the AVX512_POPCNT instruction is + * available. This is similar to the method above; see + * https://en.wikipedia.org/wiki/CPUID#EAX=7,_ECX=0:_Extended_Features + * + * Finally, we make sure the xgetbv result is consistent with the CPUID + * results. + */ +static bool +pg_popcount512_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + + /* Check for AVX512VPOPCNTDQ and AVX512F */ +#if defined(HAVE__GET_CPUID_COUNT) + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUIDEX) + __cpuidex(exx, 7, 0); +#endif + + if ((exx[2] & (0x00004000)) != 0 && (exx[1] & (0x00010000)) != 0) + { + /* + * CPUID succeeded, does the current running OS support the + * ZMM registers which are required for AVX512? This check is + * required to make sure an old OS on a new CPU is correctly + * checked or a VM hypervisor is not excluding AVX512 ZMM + * support in the VM; see "5.1.9 Detection of AVX Instructions" + * https://www.intel.com/content/www/us/en/content-details/671488/intel-64-and-ia-32-architectures-optimization-reference-manual-volume-1.html + */ + uint64 xcr = 0; +#ifdef _MSC_VER + uint64 highlow = _xgetbv(xcr); + + return (highlow & 0xE0) != 0; +#else + uint32 high; + uint32 low; + + __asm__ __volatile__("xgetbv\t\n" : "=a"(low), "=d"(high) : "c"(xcr)); + return (low & 0xE0) != 0; +#endif + } /* POPCNT 512 */ + return false; +} + /* * These functions get called on the first call to pg_popcount32 etc. * They detect whether we can use the asm implementations, and replace @@ -61,17 +108,26 @@ pg_popcount_available(void) */ static inline void set_function_pointers() { - if (pg_popcount_available()) - { + if (pg_popcount512_available()) + { /* If POPCNT512 is available, its assume that POPCNTQ is too. */ pg_popcount32 = pg_popcount32_fast; pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; + pg_popcount = pg_popcount512_fast; } else { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; + if (pg_popcount_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + pg_popcount32 = pg_popcount32_slow; + pg_popcount64 = pg_popcount64_slow; + pg_popcount = pg_popcount_slow; + } } } -- 2.34.1 ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Popcount optimization using AVX512 @ 2024-03-20 04:26 David Rowley <[email protected]> parent: Amonson, Paul D <[email protected]> 1 sibling, 1 reply; 12+ messages in thread From: David Rowley @ 2024-03-20 04:26 UTC (permalink / raw) To: Amonson, Paul D <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> On Wed, 20 Mar 2024 at 11:56, Amonson, Paul D <[email protected]> wrote: > Changed in this patch set. Thanks for rebasing. I don't think there's any need to mention Intel in each of the following comments: +# Check for Intel AVX512 intrinsics to do POPCNT calculations. +# Newer Intel processors can use AVX-512 POPCNT Capabilities (01/30/2024) AMD's Zen4 also has AVX512, so it's misleading to indicate it's an Intel only instruction. Also, writing the date isn't necessary as we have "git blame" David ^ permalink raw reply [nested|flat] 12+ messages in thread
* RE: Popcount optimization using AVX512 @ 2024-03-20 14:23 Amonson, Paul D <[email protected]> parent: David Rowley <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Amonson, Paul D @ 2024-03-20 14:23 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> > -----Original Message----- > From: David Rowley <[email protected]> > Sent: Tuesday, March 19, 2024 9:26 PM > To: Amonson, Paul D <[email protected]> > > AMD's Zen4 also has AVX512, so it's misleading to indicate it's an Intel only > instruction. Also, writing the date isn't necessary as we have "git blame" Fixed. Thanks, Paul Attachments: [application/octet-stream] v11-0001-Refactor-inlining-and-direct-calls-for-_slow-functio.patch (7.9K, ../../BL1PR11MB530495FDAC9707B800F6E59CDC332@BL1PR11MB5304.namprd11.prod.outlook.com/2-v11-0001-Refactor-inlining-and-direct-calls-for-_slow-functio.patch) download | inline diff: From e9483d2354bd58210805b17fa78091ecc007c612 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 19 Mar 2024 13:35:53 -0700 Subject: [PATCH 1/3] [Refactor] inlining and direct calls for *_slow functions. Signed-off-by: Paul Amonson <[email protected]> --- src/backend/access/heap/visibilitymap.c | 6 +- src/include/nodes/bitmapset.h | 4 +- src/include/port/pg_bitutils.h | 17 +++-- src/port/pg_bitutils.c | 90 +++++-------------------- 4 files changed, 34 insertions(+), 83 deletions(-) diff --git a/src/backend/access/heap/visibilitymap.c b/src/backend/access/heap/visibilitymap.c index 1ab6c865e3..28dc497b79 100644 --- a/src/backend/access/heap/visibilitymap.c +++ b/src/backend/access/heap/visibilitymap.c @@ -419,14 +419,14 @@ visibilitymap_count(Relation rel, BlockNumber *all_visible, BlockNumber *all_fro if (all_frozen == NULL) { for (i = 0; i < MAPSIZE / sizeof(uint64); i++) - nvisible += pg_popcount64(map[i] & VISIBLE_MASK64); + nvisible += PG_POPCOUNT64(map[i] & VISIBLE_MASK64); } else { for (i = 0; i < MAPSIZE / sizeof(uint64); i++) { - nvisible += pg_popcount64(map[i] & VISIBLE_MASK64); - nfrozen += pg_popcount64(map[i] & FROZEN_MASK64); + nvisible += PG_POPCOUNT64(map[i] & VISIBLE_MASK64); + nfrozen += PG_POPCOUNT64(map[i] & FROZEN_MASK64); } } diff --git a/src/include/nodes/bitmapset.h b/src/include/nodes/bitmapset.h index 283bea5ea9..b5631d153e 100644 --- a/src/include/nodes/bitmapset.h +++ b/src/include/nodes/bitmapset.h @@ -77,11 +77,11 @@ typedef enum #if BITS_PER_BITMAPWORD == 32 #define bmw_leftmost_one_pos(w) pg_leftmost_one_pos32(w) #define bmw_rightmost_one_pos(w) pg_rightmost_one_pos32(w) -#define bmw_popcount(w) pg_popcount32(w) +#define bmw_popcount(w) PG_POPCOUNT32(w) #elif BITS_PER_BITMAPWORD == 64 #define bmw_leftmost_one_pos(w) pg_leftmost_one_pos64(w) #define bmw_rightmost_one_pos(w) pg_rightmost_one_pos64(w) -#define bmw_popcount(w) pg_popcount64(w) +#define bmw_popcount(w) PG_POPCOUNT64(w) #else #error "invalid BITS_PER_BITMAPWORD" #endif diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 53e5239717..477e00e0da 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -300,16 +300,23 @@ pg_ceil_log2_64(uint64 num) #ifdef TRY_POPCNT_FAST /* Attempt to use the POPCNT instruction, but perform a runtime check first */ +extern int pg_popcount32_slow(uint32 word); +extern int pg_popcount64_slow(uint64 word); +extern uint64 pg_popcount_slow(const char *buf, int bytes); extern PGDLLIMPORT int (*pg_popcount32) (uint32 word); extern PGDLLIMPORT int (*pg_popcount64) (uint64 word); extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); - +#define PG_POPCOUNT32(x) pg_popcount32(x) +#define PG_POPCOUNT64(x) pg_popcount64(x) +#define PG_POPCOUNT(x,y) pg_popcount(x,y) #else /* Use a portable implementation -- no need for a function pointer. */ -extern int pg_popcount32(uint32 word); -extern int pg_popcount64(uint64 word); -extern uint64 pg_popcount(const char *buf, int bytes); - +extern int pg_popcount32_slow(uint32 word); +extern int pg_popcount64_slow(uint64 word); +extern uint64 pg_popcount_slow(const char *buf, int bytes); +#define PG_POPCOUNT32(x) pg_popcount32_slow(x) +#define PG_POPCOUNT64(x) pg_popcount64_slow(x) +#define PG_POPCOUNT(x,y) pg_popcount_slow(x,y) #endif /* TRY_POPCNT_FAST */ /* diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index 1197696e97..e629969035 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -103,9 +103,9 @@ const uint8 pg_number_of_ones[256] = { 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8 }; -static inline int pg_popcount32_slow(uint32 word); -static inline int pg_popcount64_slow(uint64 word); -static uint64 pg_popcount_slow(const char *buf, int bytes); +// static inline int pg_popcount32_slow(uint32 word); +// static inline int pg_popcount64_slow(uint64 word); +// static uint64 pg_popcount_slow(const char *buf, int bytes); #ifdef TRY_POPCNT_FAST static bool pg_popcount_available(void); @@ -119,9 +119,6 @@ static uint64 pg_popcount_fast(const char *buf, int bytes); int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; -#endif /* TRY_POPCNT_FAST */ - -#ifdef TRY_POPCNT_FAST /* * Return true if CPUID indicates that the POPCNT instruction is available. @@ -148,8 +145,7 @@ pg_popcount_available(void) * the function pointers so that subsequent calls are routed directly to * the chosen implementation. */ -static int -pg_popcount32_choose(uint32 word) +static inline void set_function_pointers() { if (pg_popcount_available()) { @@ -163,45 +159,26 @@ pg_popcount32_choose(uint32 word) pg_popcount64 = pg_popcount64_slow; pg_popcount = pg_popcount_slow; } +} +static int +pg_popcount32_choose(uint32 word) +{ + set_function_pointers(); return pg_popcount32(word); } static int pg_popcount64_choose(uint64 word) { - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - + set_function_pointers(); return pg_popcount64(word); } static uint64 pg_popcount_choose(const char *buf, int bytes) { - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - + set_function_pointers(); return pg_popcount(buf, bytes); } @@ -243,7 +220,7 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_fast * Returns the number of 1-bits in buf */ -static uint64 +static inline uint64 pg_popcount_fast(const char *buf, int bytes) { uint64 popcnt = 0; @@ -256,7 +233,7 @@ pg_popcount_fast(const char *buf, int bytes) while (bytes >= 8) { - popcnt += pg_popcount64_fast(*words++); + popcnt += PG_POPCOUNT64(*words++); bytes -= 8; } @@ -270,7 +247,7 @@ pg_popcount_fast(const char *buf, int bytes) while (bytes >= 4) { - popcnt += pg_popcount32_fast(*words++); + popcnt += PG_POPCOUNT32(*words++); bytes -= 4; } @@ -292,7 +269,7 @@ pg_popcount_fast(const char *buf, int bytes) * pg_popcount32_slow * Return the number of 1 bits set in word */ -static inline int +inline int pg_popcount32_slow(uint32 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -314,7 +291,7 @@ pg_popcount32_slow(uint32 word) * pg_popcount64_slow * Return the number of 1 bits set in word */ -static inline int +inline int pg_popcount64_slow(uint64 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -342,7 +319,7 @@ pg_popcount64_slow(uint64 word) * pg_popcount_slow * Returns the number of 1-bits in buf */ -static uint64 +uint64 pg_popcount_slow(const char *buf, int bytes) { uint64 popcnt = 0; @@ -383,36 +360,3 @@ pg_popcount_slow(const char *buf, int bytes) return popcnt; } - -#ifndef TRY_POPCNT_FAST - -/* - * When the POPCNT instruction is not available, there's no point in using - * function pointers to vary the implementation between the fast and slow - * method. We instead just make these actual external functions when - * TRY_POPCNT_FAST is not defined. The compiler should be able to inline - * the slow versions here. - */ -int -pg_popcount32(uint32 word) -{ - return pg_popcount32_slow(word); -} - -int -pg_popcount64(uint64 word) -{ - return pg_popcount64_slow(word); -} - -/* - * pg_popcount - * Returns the number of 1-bits in buf - */ -uint64 -pg_popcount(const char *buf, int bytes) -{ - return pg_popcount_slow(buf, bytes); -} - -#endif /* !TRY_POPCNT_FAST */ -- 2.34.1 [application/octet-stream] v11-0002-Refactor-Seperated-slow-fast-and-choose-functionalit.patch (15.8K, ../../BL1PR11MB530495FDAC9707B800F6E59CDC332@BL1PR11MB5304.namprd11.prod.outlook.com/3-v11-0002-Refactor-Seperated-slow-fast-and-choose-functionalit.patch) download | inline diff: From e5e4cac323b913e9fcdbd17d6b07316a21f7ff5c Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 19 Mar 2024 13:37:31 -0700 Subject: [PATCH 2/3] [Refactor] Seperated slow, fast, and choose functionality into files. Signed-off-by: Paul Amonson <[email protected]> --- contrib/intarray/_intbig_gist.c | 2 +- contrib/ltree/_ltree_gist.c | 2 +- contrib/pageinspect/heapfuncs.c | 4 +- contrib/pg_trgm/trgm_gist.c | 2 +- contrib/pg_walinspect/pg_walinspect.c | 2 +- src/backend/lib/bloomfilter.c | 2 +- src/backend/postmaster/syslogger.c | 2 +- src/backend/utils/adt/tsgistidx.c | 2 +- src/backend/utils/adt/varbit.c | 2 +- src/backend/utils/adt/varlena.c | 2 +- src/port/Makefile | 2 + src/port/meson.build | 2 + src/port/pg_bitutils.c | 171 +------------------------- src/port/pg_popcount_x86_64_accel.c | 101 +++++++++++++++ src/port/pg_popcount_x86_64_choose.c | 98 +++++++++++++++ 15 files changed, 215 insertions(+), 181 deletions(-) create mode 100644 src/port/pg_popcount_x86_64_accel.c create mode 100644 src/port/pg_popcount_x86_64_choose.c diff --git a/contrib/intarray/_intbig_gist.c b/contrib/intarray/_intbig_gist.c index 9699fbf3b4..a12ea7ed9b 100644 --- a/contrib/intarray/_intbig_gist.c +++ b/contrib/intarray/_intbig_gist.c @@ -210,7 +210,7 @@ g_intbig_compress(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount(sign, siglen); + return PG_POPCOUNT(sign, siglen); } static int diff --git a/contrib/ltree/_ltree_gist.c b/contrib/ltree/_ltree_gist.c index e89a39a5b5..bd66ec2e65 100644 --- a/contrib/ltree/_ltree_gist.c +++ b/contrib/ltree/_ltree_gist.c @@ -180,7 +180,7 @@ _ltree_union(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount((const char *) sign, siglen); + return PG_POPCOUNT((const char *) sign, siglen); } static int diff --git a/contrib/pageinspect/heapfuncs.c b/contrib/pageinspect/heapfuncs.c index 3faeabc711..089842962f 100644 --- a/contrib/pageinspect/heapfuncs.c +++ b/contrib/pageinspect/heapfuncs.c @@ -527,8 +527,8 @@ heap_tuple_infomask_flags(PG_FUNCTION_ARGS) if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) elog(ERROR, "return type must be a row type"); - bitcnt = pg_popcount((const char *) &t_infomask, sizeof(uint16)) + - pg_popcount((const char *) &t_infomask2, sizeof(uint16)); + bitcnt = PG_POPCOUNT((const char *) &t_infomask, sizeof(uint16)) + + PG_POPCOUNT((const char *) &t_infomask2, sizeof(uint16)); /* If no flags, return a set of empty arrays */ if (bitcnt <= 0) diff --git a/contrib/pg_trgm/trgm_gist.c b/contrib/pg_trgm/trgm_gist.c index 9ef2e38560..850316196f 100644 --- a/contrib/pg_trgm/trgm_gist.c +++ b/contrib/pg_trgm/trgm_gist.c @@ -648,7 +648,7 @@ gtrgm_same(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount(sign, siglen); + return PG_POPCOUNT(sign, siglen); } static int diff --git a/contrib/pg_walinspect/pg_walinspect.c b/contrib/pg_walinspect/pg_walinspect.c index ee2918726d..93a7b4842a 100644 --- a/contrib/pg_walinspect/pg_walinspect.c +++ b/contrib/pg_walinspect/pg_walinspect.c @@ -303,7 +303,7 @@ GetWALBlockInfo(FunctionCallInfo fcinfo, XLogReaderState *record, block_fpi_len = blk->bimg_len; /* Construct and save block_fpi_info */ - bitcnt = pg_popcount((const char *) &blk->bimg_info, + bitcnt = PG_POPCOUNT((const char *) &blk->bimg_info, sizeof(uint8)); flags = (Datum *) palloc0(sizeof(Datum) * bitcnt); if ((blk->bimg_info & BKPIMAGE_HAS_HOLE) != 0) diff --git a/src/backend/lib/bloomfilter.c b/src/backend/lib/bloomfilter.c index 360d21ca45..c01b069c01 100644 --- a/src/backend/lib/bloomfilter.c +++ b/src/backend/lib/bloomfilter.c @@ -187,7 +187,7 @@ double bloom_prop_bits_set(bloom_filter *filter) { int bitset_bytes = filter->m / BITS_PER_BYTE; - uint64 bits_set = pg_popcount((char *) filter->bitset, bitset_bytes); + uint64 bits_set = PG_POPCOUNT((char *) filter->bitset, bitset_bytes); return bits_set / (double) filter->m; } diff --git a/src/backend/postmaster/syslogger.c b/src/backend/postmaster/syslogger.c index 08efe74cc9..85c57b3154 100644 --- a/src/backend/postmaster/syslogger.c +++ b/src/backend/postmaster/syslogger.c @@ -898,7 +898,7 @@ process_pipe_input(char *logbuffer, int *bytes_in_logbuffer) if (p.nuls[0] == '\0' && p.nuls[1] == '\0' && p.len > 0 && p.len <= PIPE_MAX_PAYLOAD && p.pid != 0 && - pg_popcount((char *) &dest_flags, 1) == 1) + PG_POPCOUNT((char *) &dest_flags, 1) == 1) { List *buffer_list; ListCell *cell; diff --git a/src/backend/utils/adt/tsgistidx.c b/src/backend/utils/adt/tsgistidx.c index 5698ee5502..d7a76faf31 100644 --- a/src/backend/utils/adt/tsgistidx.c +++ b/src/backend/utils/adt/tsgistidx.c @@ -489,7 +489,7 @@ gtsvector_same(PG_FUNCTION_ARGS) static int32 sizebitvec(BITVECP sign, int siglen) { - return pg_popcount(sign, siglen); + return PG_POPCOUNT(sign, siglen); } static int diff --git a/src/backend/utils/adt/varbit.c b/src/backend/utils/adt/varbit.c index 8fcf3fb731..3f287cd54d 100644 --- a/src/backend/utils/adt/varbit.c +++ b/src/backend/utils/adt/varbit.c @@ -1212,7 +1212,7 @@ bit_bit_count(PG_FUNCTION_ARGS) { VarBit *arg = PG_GETARG_VARBIT_P(0); - PG_RETURN_INT64(pg_popcount((char *) VARBITS(arg), VARBITBYTES(arg))); + PG_RETURN_INT64(PG_POPCOUNT((char *) VARBITS(arg), VARBITBYTES(arg))); } /* diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 8d28dd42ce..809e6a59ab 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -3151,7 +3151,7 @@ bytea_bit_count(PG_FUNCTION_ARGS) { bytea *t1 = PG_GETARG_BYTEA_PP(0); - PG_RETURN_INT64(pg_popcount(VARDATA_ANY(t1), VARSIZE_ANY_EXHDR(t1))); + PG_RETURN_INT64(PG_POPCOUNT(VARDATA_ANY(t1), VARSIZE_ANY_EXHDR(t1))); } /* diff --git a/src/port/Makefile b/src/port/Makefile index dcc8737e68..1499985dfc 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -44,6 +44,8 @@ OBJS = \ noblock.o \ path.o \ pg_bitutils.o \ + pg_popcount_x86_64_choose.o \ + pg_popcount_x86_64_accel.o \ pg_strong_random.o \ pgcheckdir.o \ pgmkdirp.o \ diff --git a/src/port/meson.build b/src/port/meson.build index 92b593e6ef..cf6e9fa06c 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -7,6 +7,8 @@ pgport_sources = [ 'noblock.c', 'path.c', 'pg_bitutils.c', + 'pg_popcount_x86_64_choose.c', + 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index e629969035..f08820b35b 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -11,14 +11,6 @@ *------------------------------------------------------------------------- */ #include "c.h" - -#ifdef HAVE__GET_CPUID -#include <cpuid.h> -#endif -#ifdef HAVE__CPUID -#include <intrin.h> -#endif - #include "port/pg_bitutils.h" @@ -103,167 +95,6 @@ const uint8 pg_number_of_ones[256] = { 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8 }; -// static inline int pg_popcount32_slow(uint32 word); -// static inline int pg_popcount64_slow(uint64 word); -// static uint64 pg_popcount_slow(const char *buf, int bytes); - -#ifdef TRY_POPCNT_FAST -static bool pg_popcount_available(void); -static int pg_popcount32_choose(uint32 word); -static int pg_popcount64_choose(uint64 word); -static uint64 pg_popcount_choose(const char *buf, int bytes); -static inline int pg_popcount32_fast(uint32 word); -static inline int pg_popcount64_fast(uint64 word); -static uint64 pg_popcount_fast(const char *buf, int bytes); - -int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; -int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; -uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; - -/* - * Return true if CPUID indicates that the POPCNT instruction is available. - */ -static bool -pg_popcount_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; - -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); -#else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 23)) != 0; /* POPCNT */ -} - -/* - * These functions get called on the first call to pg_popcount32 etc. - * They detect whether we can use the asm implementations, and replace - * the function pointers so that subsequent calls are routed directly to - * the chosen implementation. - */ -static inline void set_function_pointers() -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } -} - -static int -pg_popcount32_choose(uint32 word) -{ - set_function_pointers(); - return pg_popcount32(word); -} - -static int -pg_popcount64_choose(uint64 word) -{ - set_function_pointers(); - return pg_popcount64(word); -} - -static uint64 -pg_popcount_choose(const char *buf, int bytes) -{ - set_function_pointers(); - return pg_popcount(buf, bytes); -} - -/* - * pg_popcount32_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount32_fast(uint32 word) -{ -#ifdef _MSC_VER - return __popcnt(word); -#else - uint32 res; - -__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif -} - -/* - * pg_popcount64_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount64_fast(uint64 word) -{ -#ifdef _MSC_VER - return __popcnt64(word); -#else - uint64 res; - -__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif -} - -/* - * pg_popcount_fast - * Returns the number of 1-bits in buf - */ -static inline uint64 -pg_popcount_fast(const char *buf, int bytes) -{ - uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) - { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += PG_POPCOUNT64(*words++); - bytes -= 8; - } - - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += PG_POPCOUNT32(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif - - /* Process any remaining bytes */ - while (bytes--) - popcnt += pg_number_of_ones[(unsigned char) *buf++]; - - return popcnt; -} - -#endif /* TRY_POPCNT_FAST */ - /* * pg_popcount32_slow @@ -319,7 +150,7 @@ pg_popcount64_slow(uint64 word) * pg_popcount_slow * Returns the number of 1-bits in buf */ -uint64 +inline uint64 pg_popcount_slow(const char *buf, int bytes) { uint64 popcnt = 0; diff --git a/src/port/pg_popcount_x86_64_accel.c b/src/port/pg_popcount_x86_64_accel.c new file mode 100644 index 0000000000..d63e8aa30f --- /dev/null +++ b/src/port/pg_popcount_x86_64_accel.c @@ -0,0 +1,101 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_accel.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_accel.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" +#include "port/pg_bitutils.h" + +#ifdef TRY_POPCNT_FAST +int pg_popcount32_fast(uint32 word); +int pg_popcount64_fast(uint64 word); +uint64 pg_popcount_fast(const char *buf, int bytes); + +/* + * pg_popcount32_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount32_fast(uint32 word) +{ +#ifdef _MSC_VER + return __popcnt(word); +#else + uint32 res; + +__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount64_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount64_fast(uint64 word) +{ +#ifdef _MSC_VER + return __popcnt64(word); +#else + uint64 res; + +__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount_fast + * Returns the number of 1-bits in buf + */ +uint64 +pg_popcount_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + +#if SIZEOF_VOID_P >= 8 + /* Process in 64-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(8, buf)) + { + const uint64 *words = (const uint64 *) buf; + + while (bytes >= 8) + { + popcnt += PG_POPCOUNT64(*words++); + bytes -= 8; + } + + buf = (const char *) words; + } +#else + /* Process in 32-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(4, buf)) + { + const uint32 *words = (const uint32 *) buf; + + while (bytes >= 4) + { + popcnt += PG_POPCOUNT32(*words++); + bytes -= 4; + } + + buf = (const char *) words; + } +#endif + + /* Process any remaining bytes */ + while (bytes--) + popcnt += pg_number_of_ones[(unsigned char) *buf++]; + + return popcnt; +} + +#endif /* TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_choose.c b/src/port/pg_popcount_x86_64_choose.c new file mode 100644 index 0000000000..1a0022a0b3 --- /dev/null +++ b/src/port/pg_popcount_x86_64_choose.c @@ -0,0 +1,98 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_choose.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_choose.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" + +#ifdef HAVE__GET_CPUID +#include <cpuid.h> +#endif +#ifdef HAVE__CPUID +#include <intrin.h> +#endif + +#include "port/pg_bitutils.h" + +#ifdef TRY_POPCNT_FAST +int pg_popcount32_fast(uint32 word); +int pg_popcount64_fast(uint64 word); +uint64 pg_popcount_fast(const char *buf, int bytes); + +static int pg_popcount32_choose(uint32 word); +static int pg_popcount64_choose(uint64 word); +static uint64 pg_popcount_choose(const char *buf, int bytes); + +int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; +int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; +uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; + +/* + * Return true if CPUID indicates that the POPCNT instruction is available. + */ +static bool +pg_popcount_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + +#if defined(HAVE__GET_CPUID) + __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUID) + __cpuid(exx, 1); +#else +#error cpuid instruction not available +#endif + + return (exx[2] & (1 << 23)) != 0; /* POPCNT */ +} + +/* + * These functions get called on the first call to pg_popcount32 etc. + * They detect whether we can use the asm implementations, and replace + * the function pointers so that subsequent calls are routed directly to + * the chosen implementation. + */ +static inline void set_function_pointers() +{ + if (pg_popcount_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + pg_popcount32 = pg_popcount32_slow; + pg_popcount64 = pg_popcount64_slow; + pg_popcount = pg_popcount_slow; + } +} + +static int +pg_popcount32_choose(uint32 word) +{ + set_function_pointers(); + return pg_popcount32(word); +} + +static int +pg_popcount64_choose(uint64 word) +{ + set_function_pointers(); + return pg_popcount64(word); +} + +static uint64 +pg_popcount_choose(const char *buf, int bytes) +{ + set_function_pointers(); + return pg_popcount(buf, bytes); +} +#endif /* TRY_POPCNT_FAST */ -- 2.34.1 [application/octet-stream] v11-0003-Feature-Add-POPCNT512-accelerated-functionality-for-.patch (23.5K, ../../BL1PR11MB530495FDAC9707B800F6E59CDC332@BL1PR11MB5304.namprd11.prod.outlook.com/4-v11-0003-Feature-Add-POPCNT512-accelerated-functionality-for-.patch) download | inline diff: From b586e558d8eec0f5d5c45dfabec317ad5300e8d6 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Tue, 19 Mar 2024 13:40:04 -0700 Subject: [PATCH 3/3] [Feature] Add POPCNT512 accelerated functionality for x86_64. Signed-off-by: Paul Amonson <[email protected]> --- config/c-compiler.m4 | 37 +++++ configure | 205 +++++++++++++++++++++++++++ configure.ac | 44 ++++++ meson.build | 72 ++++++++++ src/Makefile.global.in | 1 + src/backend/commands/user.c | 2 +- src/include/pg_config.h.in | 12 ++ src/include/port/pg_bitutils.h | 6 +- src/makefiles/meson.build | 1 + src/port/Makefile | 5 + src/port/meson.build | 6 +- src/port/pg_popcount_x86_64_accel.c | 33 +++++ src/port/pg_popcount_x86_64_choose.c | 68 ++++++++- 13 files changed, 479 insertions(+), 13 deletions(-) diff --git a/config/c-compiler.m4 b/config/c-compiler.m4 index 3268a780bb..94e3e713aa 100644 --- a/config/c-compiler.m4 +++ b/config/c-compiler.m4 @@ -694,3 +694,40 @@ if test x"$Ac_cachevar" = x"yes"; then fi undefine([Ac_cachevar])dnl ])# PGAC_LOONGARCH_CRC32C_INTRINSICS + +# PGAC_AVX512_POPCNT_INTRINSICS +# --------------------------- +# Check if the compiler supports the x86_64 AVX512 POPCNT instructions using +# intrinsics used in CPUID features AVX512F and AVX512VPOPCNTDQ. +# +# Optional compiler flags can be passed as argument (e.g. -mavx512vpopcntdq). +# If the intrinsics are supported then pgac_avx512_popcnt_intrinsics and +# CFLAGS_AVX512_POPCNT are set. +AC_DEFUN([PGAC_AVX512_POPCNT_INTRINSICS], +[define([Ac_cachevar], [AS_TR_SH([pgac_cv_avx512_popcnt_intrinsics_$1])])dnl +AC_CACHE_CHECK([for _mm512_popcnt_epi64 with CFLAGS=$1], [Ac_cachevar], +[pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS $1" +AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h>], + [const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0;])], + [Ac_cachevar=yes], + [Ac_cachevar=no]) +CFLAGS="$pgac_save_CFLAGS"]) +if test x"$Ac_cachevar" = x"yes"; then + CFLAGS_AVX512_POPCNT="$1" + pgac_avx512_popcnt_intrinsics=yes +fi +undefine([Ac_cachevar])dnl +])# PGAC_AVX512_POPCNT_INTRINSICS diff --git a/configure b/configure index 36feeafbb2..0fbfc7c78f 100755 --- a/configure +++ b/configure @@ -647,6 +647,7 @@ MSGFMT_FLAGS MSGFMT PG_CRC32C_OBJS CFLAGS_CRC +CFLAGS_AVX512_POPCNT LIBOBJS OPENSSL ZSTD @@ -17404,6 +17405,41 @@ $as_echo "#define HAVE__GET_CPUID 1" >>confdefs.h fi +# Check for x86 cpuid_count instruction +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __get_cpuid_count" >&5 +$as_echo_n "checking for __get_cpuid_count... " >&6; } +if ${pgac_cv__get_cpuid_count+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <cpuid.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__get_cpuid_count="yes" +else + pgac_cv__get_cpuid_count="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__get_cpuid_count" >&5 +$as_echo "$pgac_cv__get_cpuid_count" >&6; } +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + +$as_echo "#define HAVE__GET_CPUID_COUNT 1" >>confdefs.h + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuid" >&5 $as_echo_n "checking for __cpuid... " >&6; } if ${pgac_cv__cpuid+:} false; then : @@ -17438,6 +17474,175 @@ $as_echo "#define HAVE__CPUID 1" >>confdefs.h fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuidex" >&5 +$as_echo_n "checking for __cpuidex... " >&6; } +if ${pgac_cv__cpuidex+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <intrin.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__cpuidex="yes" +else + pgac_cv__cpuidex="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__cpuidex" >&5 +$as_echo "$pgac_cv__cpuidex" >&6; } +if test x"$pgac_cv__cpuidex" = x"yes"; then + +$as_echo "#define HAVE__CPUIDEX 1" >>confdefs.h + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __immintrin" >&5 +$as_echo_n "checking for __immintrin... " >&6; } +if ${pgac_cv__immintrin+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +int +main () +{ +/* Don't exclude code so added return. */ + return 1701; + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__immintrin="yes" +else + pgac_cv__immintrin="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__immintrin" >&5 +$as_echo "$pgac_cv__immintrin" >&6; } +if test x"$pgac_cv__immintrin" = x"yes"; then + +$as_echo "#define HAVE__IMMINTRIN 1" >>confdefs.h + +fi + +# Check for AVX512 intrinsics to do POPCNT calculations. +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics_+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS " +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics_=yes +else + pgac_cv_avx512_popcnt_intrinsics_=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics_" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics_" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics_" = x"yes"; then + CFLAGS_AVX512_POPCNT="" + pgac_avx512_popcnt_intrinsics=yes +fi + +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS -mavx512vpopcntdq" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=yes +else + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" = x"yes"; then + CFLAGS_AVX512_POPCNT="-mavx512vpopcntdq" + pgac_avx512_popcnt_intrinsics=yes +fi + + +$as_echo "#define HAVE__AVX512_POPCNT 1" >>confdefs.h + +fi + + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/configure.ac b/configure.ac index 57f734879e..3c741d457d 100644 --- a/configure.ac +++ b/configure.ac @@ -2052,6 +2052,18 @@ if test x"$pgac_cv__get_cpuid" = x"yes"; then AC_DEFINE(HAVE__GET_CPUID, 1, [Define to 1 if you have __get_cpuid.]) fi +# Check for x86 cpuid_count instruction +AC_CACHE_CHECK([for __get_cpuid_count], [pgac_cv__get_cpuid_count], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <cpuid.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + ]])], + [pgac_cv__get_cpuid_count="yes"], + [pgac_cv__get_cpuid_count="no"])]) +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + AC_DEFINE(HAVE__GET_CPUID_COUNT, 1, [Define to 1 if you have __get_cpuid.]) +fi + AC_CACHE_CHECK([for __cpuid], [pgac_cv__cpuid], [AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], [[unsigned int exx[4] = {0, 0, 0, 0}; @@ -2063,6 +2075,38 @@ if test x"$pgac_cv__cpuid" = x"yes"; then AC_DEFINE(HAVE__CPUID, 1, [Define to 1 if you have __cpuid.]) fi +AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + ]])], + [pgac_cv__cpuidex="yes"], + [pgac_cv__cpuidex="no"])]) +if test x"$pgac_cv__cpuidex" = x"yes"; then + AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.]) +fi + +AC_CACHE_CHECK([for __immintrin], [pgac_cv__immintrin], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h>], + [[/* Don't exclude code so added return. */ + return 1701; + ]])], + [pgac_cv__immintrin="yes"], + [pgac_cv__immintrin="no"])]) +if test x"$pgac_cv__immintrin" = x"yes"; then + AC_DEFINE(HAVE__IMMINTRIN, 1, [Define to 1 if you have immintrin.]) +fi + +# Check for AVX512 intrinsics to do POPCNT calculations. +# +PGAC_AVX512_POPCNT_INTRINSICS([]) +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + PGAC_AVX512_POPCNT_INTRINSICS([-mavx512vpopcntdq]) + AC_DEFINE(HAVE__AVX512_POPCNT, 1, [Define to 1 if you have cpu + support for AVX512 POPCNT.]) +fi +AC_SUBST(CFLAGS_AVX512_POPCNT) + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/meson.build b/meson.build index c8fdfeb0ec..d661405fee 100644 --- a/meson.build +++ b/meson.build @@ -1783,6 +1783,37 @@ elif cc.links(''' endif +if cc.links(''' + #include <cpuid.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + } + ''', name: '__get_cpuid_count', + args: test_c_args) + cdata.set('HAVE__GET_CPUID_COUNT', 1) +elif cc.links(''' + #include <intrin.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __cpuidex(exx, 7, 0); + } + ''', name: '__cpuidex', + args: test_c_args) + cdata.set('HAVE__CPUIDEX', 1) +endif + + +# Check for header immintrin.h +if cc.has_header('immintrin.h', + include_directories: postgres_inc, args: test_c_args) + cdata.set('HAVE__IMMINTRIN', 1, + description: 'Define to 1 if you have the immintrin.h header file.') +endif + + # Defend against clang being used on x86-32 without SSE2 enabled. As current # versions of clang do not understand -fexcess-precision=standard, the use of # x87 floating point operations leads to problems like isinf possibly returning @@ -2157,6 +2188,47 @@ elif host_cpu == 'ppc' or host_cpu == 'ppc64' endif +############################################################### +# AVX 512 POPCNT Intrinsic check +############################################################### +have_avx512_popcnt = false +cflags_avx512_popcnt = [] +if host_cpu == 'x86_64' + test_flags = ['-mavx512vpopcntdq'] + if host_system == 'windows' + test_flags = ['/arch:AVX512'] + endif + prog = ''' + #include <immintrin.h> + #include <stdint.h> + #include <stdlib.h> + #include <string.h> + void main(void) + { + const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + }''' + if cc.links(prog, name: '_mm512_* methods with -mavx512vpopcntdq flag.', + args: test_c_args + test_flags) + have_avx512_popcnt = true + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cdata.set('HAVE__AVX512_POPCNT', 1) + cflags_avx512_popcnt = test_flags + else + have_avx512_popcnt = false + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cflags_avx512_popcnt = [] + endif # compile/link test +endif # host_cpu check + ############################################################### # Library / OS tests diff --git a/src/Makefile.global.in b/src/Makefile.global.in index 8b3f8c24e0..089f49b7f3 100644 --- a/src/Makefile.global.in +++ b/src/Makefile.global.in @@ -263,6 +263,7 @@ CXXFLAGS_SL_MODULE = @CXXFLAGS_SL_MODULE@ CFLAGS_UNROLL_LOOPS = @CFLAGS_UNROLL_LOOPS@ CFLAGS_VECTORIZE = @CFLAGS_VECTORIZE@ CFLAGS_CRC = @CFLAGS_CRC@ +CFLAGS_AVX512_POPCNT = @CFLAGS_AVX512_POPCNT@ PERMIT_DECLARATION_AFTER_STATEMENT = @PERMIT_DECLARATION_AFTER_STATEMENT@ CXXFLAGS = @CXXFLAGS@ diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c index c75cde2e8e..77d72daa87 100644 --- a/src/backend/commands/user.c +++ b/src/backend/commands/user.c @@ -2330,7 +2330,7 @@ plan_single_revoke(CatCList *memlist, RevokeRoleGrantAction *actions, * wouldn't work properly if such syntax were added, so assert that our * caller isn't trying to do that. */ - Assert(pg_popcount32(popt->specified) <= 1); + Assert(PG_POPCOUNT32(popt->specified) <= 1); for (i = 0; i < memlist->n_members; ++i) { diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 591e1ca3df..33a831e768 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -558,6 +558,18 @@ /* Define to 1 if you have __get_cpuid. */ #undef HAVE__GET_CPUID +/* Define to 1 if you have __get_cpuid_count. */ +#undef HAVE__GET_CPUID_COUNT + +/* Define to 1 if you have __get_cpuidex. */ +#undef HAVE__GET_CPUIDEX + +/* Define to 1 if you have immintrin. */ +#undef HAVE__IMMINTRIN + +/* Define to 1 if you have AVX512. */ +#undef HAVE__AVX512_POPCNT + /* Define to 1 if your compiler understands _Static_assert. */ #undef HAVE__STATIC_ASSERT diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 477e00e0da..ead4f79def 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -300,9 +300,9 @@ pg_ceil_log2_64(uint64 num) #ifdef TRY_POPCNT_FAST /* Attempt to use the POPCNT instruction, but perform a runtime check first */ -extern int pg_popcount32_slow(uint32 word); -extern int pg_popcount64_slow(uint64 word); -extern uint64 pg_popcount_slow(const char *buf, int bytes); +extern inline int pg_popcount32_slow(uint32 word); +extern inline int pg_popcount64_slow(uint64 word); +extern inline uint64 pg_popcount_slow(const char *buf, int bytes); extern PGDLLIMPORT int (*pg_popcount32) (uint32 word); extern PGDLLIMPORT int (*pg_popcount64) (uint64 word); extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build index b0f4178b3d..ee3647282e 100644 --- a/src/makefiles/meson.build +++ b/src/makefiles/meson.build @@ -100,6 +100,7 @@ pgxs_kv = { ' '.join(cflags_no_decl_after_statement), 'CFLAGS_CRC': ' '.join(cflags_crc), + 'CFLAGS_AVX512_POPCNT': ' '.join(cflags_avx512_popcnt), 'CFLAGS_UNROLL_LOOPS': ' '.join(unroll_loops_cflags), 'CFLAGS_VECTORIZE': ' '.join(vectorize_cflags), diff --git a/src/port/Makefile b/src/port/Makefile index 1499985dfc..66ef151565 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -89,6 +89,11 @@ pg_crc32c_sse42.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_shlib.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_srv.o: CFLAGS+=$(CFLAGS_CRC) +# Newer processors can use AVX-512 POPCNT Capabilities +pg_popcount_x86_64_accel.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_shlib.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_srv.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) + # all versions of pg_crc32c_armv8.o need CFLAGS_CRC pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC) diff --git a/src/port/meson.build b/src/port/meson.build index cf6e9fa06c..0647e7a4f7 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,7 +8,6 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_x86_64_choose.c', - 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', @@ -86,6 +85,7 @@ replace_funcs_pos = [ ['pg_crc32c_sse42', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK', 'crc'], ['pg_crc32c_sse42_choose', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], ['pg_crc32c_sb8', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], + ['pg_popcount_x86_64_accel', 'USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 'avx512'], # arm / aarch64 ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C'], @@ -100,8 +100,8 @@ replace_funcs_pos = [ ['pg_crc32c_sb8', 'USE_SLICING_BY_8_CRC32C'], ] -pgport_cflags = {'crc': cflags_crc} -pgport_sources_cflags = {'crc': []} +pgport_cflags = {'crc': cflags_crc, 'avx512': cflags_avx512_popcnt} +pgport_sources_cflags = {'crc': [], 'avx512': []} foreach f : replace_funcs_neg func = f.get(0) diff --git a/src/port/pg_popcount_x86_64_accel.c b/src/port/pg_popcount_x86_64_accel.c index d63e8aa30f..f65184ddce 100644 --- a/src/port/pg_popcount_x86_64_accel.c +++ b/src/port/pg_popcount_x86_64_accel.c @@ -13,10 +13,15 @@ #include "c.h" #include "port/pg_bitutils.h" +#if defined(HAVE__IMMINTRIN) +#include <immintrin.h> +#endif + #ifdef TRY_POPCNT_FAST int pg_popcount32_fast(uint32 word); int pg_popcount64_fast(uint64 word); uint64 pg_popcount_fast(const char *buf, int bytes); +uint64 pg_popcount512_fast(const char *buf, int bytes); /* * pg_popcount32_fast @@ -98,4 +103,32 @@ pg_popcount_fast(const char *buf, int bytes) return popcnt; } +/* + * Use AVX-512 Intrinsics for supported CPUs or fall back the non-152 fast + * implem entation and use the best 64 bit fast methods. If no fast + * methods are used this will fall back to __builtin_* or pure software. + */ +uint64 +pg_popcount512_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + #if defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 + __m512i accumulator = _mm512_setzero_si512(); + + while (bytes >= 64) + { + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + + accumulator = _mm512_add_epi64(accumulator, p); + bytes -= 64; + buf += 64; + } + + popcnt = _mm512_reduce_add_epi64(accumulator); +#endif /* defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 */ + + /* Process any remaining bytes */ + return popcnt + pg_popcount_fast(buf, bytes); +} #endif /* TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_choose.c b/src/port/pg_popcount_x86_64_choose.c index 1a0022a0b3..d25d251e9b 100644 --- a/src/port/pg_popcount_x86_64_choose.c +++ b/src/port/pg_popcount_x86_64_choose.c @@ -25,6 +25,7 @@ int pg_popcount32_fast(uint32 word); int pg_popcount64_fast(uint64 word); uint64 pg_popcount_fast(const char *buf, int bytes); +uint64 pg_popcount512_fast(const char *buf, int bytes); static int pg_popcount32_choose(uint32 word); static int pg_popcount64_choose(uint64 word); @@ -53,6 +54,52 @@ pg_popcount_available(void) return (exx[2] & (1 << 23)) != 0; /* POPCNT */ } +/* + * Return true if CPUID indicates that the AVX512_POPCNT instruction is + * available. This is similar to the method above; see + * https://en.wikipedia.org/wiki/CPUID#EAX=7,_ECX=0:_Extended_Features + * + * Finally, we make sure the xgetbv result is consistent with the CPUID + * results. + */ +static bool +pg_popcount512_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + + /* Check for AVX512VPOPCNTDQ and AVX512F */ +#if defined(HAVE__GET_CPUID_COUNT) + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUIDEX) + __cpuidex(exx, 7, 0); +#endif + + if ((exx[2] & (0x00004000)) != 0 && (exx[1] & (0x00010000)) != 0) + { + /* + * CPUID succeeded, does the current running OS support the + * ZMM registers which are required for AVX512? This check is + * required to make sure an old OS on a new CPU is correctly + * checked or a VM hypervisor is not excluding AVX512 ZMM + * support in the VM; see "5.1.9 Detection of AVX Instructions" + * https://www.intel.com/content/www/us/en/content-details/671488/intel-64-and-ia-32-architectures-optimization-reference-manual-volume-1.html + */ + uint64 xcr = 0; +#ifdef _MSC_VER + uint64 highlow = _xgetbv(xcr); + + return (highlow & 0xE0) != 0; +#else + uint32 high; + uint32 low; + + __asm__ __volatile__("xgetbv\t\n" : "=a"(low), "=d"(high) : "c"(xcr)); + return (low & 0xE0) != 0; +#endif + } /* POPCNT 512 */ + return false; +} + /* * These functions get called on the first call to pg_popcount32 etc. * They detect whether we can use the asm implementations, and replace @@ -61,17 +108,26 @@ pg_popcount_available(void) */ static inline void set_function_pointers() { - if (pg_popcount_available()) - { + if (pg_popcount512_available()) + { /* If POPCNT512 is available, its assume that POPCNTQ is too. */ pg_popcount32 = pg_popcount32_fast; pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; + pg_popcount = pg_popcount512_fast; } else { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; + if (pg_popcount_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + pg_popcount32 = pg_popcount32_slow; + pg_popcount64 = pg_popcount64_slow; + pg_popcount = pg_popcount_slow; + } } } -- 2.34.1 ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Popcount optimization using AVX512 @ 2024-03-21 00:28 David Rowley <[email protected]> parent: Amonson, Paul D <[email protected]> 1 sibling, 1 reply; 12+ messages in thread From: David Rowley @ 2024-03-21 00:28 UTC (permalink / raw) To: Amonson, Paul D <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> On Wed, 20 Mar 2024 at 11:56, Amonson, Paul D <[email protected]> wrote: > Changed in this patch set. > > * Rebased. > * Direct *slow* calls via macros as shown in example patch. > * Changed the choose filename to be platform specific as suggested. > * Falls back to intermediate "Fast" methods if AVX512 is not available at runtime. > * inline used where is makes sense, remember using "extern" negates "inline". I'm not sure about this "extern negates inline" comment. It seems to me the compiler is perfectly free to inline a static function into an external function and it's free to inline the static function elsewhere within the same .c file. The final sentence of the following comment that the 0001 patch removes explains this: /* * When the POPCNT instruction is not available, there's no point in using * function pointers to vary the implementation between the fast and slow * method. We instead just make these actual external functions when * TRY_POPCNT_FAST is not defined. The compiler should be able to inline * the slow versions here. */ Also, have a look at [1]. You'll see f_slow() wasn't even compiled and the code was just inlined into f(). I just added the __attribute__((noinline)) so that usage() wouldn't just perform constant folding and just return 6. I think, unless you have evidence that some common compiler isn't inlining the static into the extern then we shouldn't add the macros. It adds quite a bit of churn to the patch and will break out of core code as you no longer have functions named pg_popcount32(), pg_popcount64() and pg_popcount(). David [1] https://godbolt.org/z/6joExb79d ^ permalink raw reply [nested|flat] 12+ messages in thread
* RE: Popcount optimization using AVX512 @ 2024-03-21 19:17 Amonson, Paul D <[email protected]> parent: David Rowley <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Amonson, Paul D @ 2024-03-21 19:17 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> > -----Original Message----- > From: David Rowley <[email protected]> > Sent: Wednesday, March 20, 2024 5:28 PM > To: Amonson, Paul D <[email protected]> > Cc: Nathan Bossart <[email protected]>; Andres Freund > > I'm not sure about this "extern negates inline" comment. It seems to me the > compiler is perfectly free to inline a static function into an external function > and it's free to inline the static function elsewhere within the same .c file. > > The final sentence of the following comment that the 0001 patch removes > explains this: > > /* > * When the POPCNT instruction is not available, there's no point in using > * function pointers to vary the implementation between the fast and slow > * method. We instead just make these actual external functions when > * TRY_POPCNT_FAST is not defined. The compiler should be able to inline > * the slow versions here. > */ > > Also, have a look at [1]. You'll see f_slow() wasn't even compiled and the code > was just inlined into f(). I just added the > __attribute__((noinline)) so that usage() wouldn't just perform constant > folding and just return 6. > > I think, unless you have evidence that some common compiler isn't inlining the > static into the extern then we shouldn't add the macros. > It adds quite a bit of churn to the patch and will break out of core code as you > no longer have functions named pg_popcount32(), > pg_popcount64() and pg_popcount(). This may be a simple misunderstanding extern != static. If I use the "extern" keyword then a symbol *will* be generated and inline will be ignored. This is NOT true of "static inline", where the compiler will try to inline the method. :) In this patch set: * I removed the macro implementation. * Made everything that could possibly be inlined marked with the "static inline" keyword. * Conditionally made the *_slow() functions "static inline" when TRY_POPCONT_FAST is not set. * Found and fixed some whitespace errors in the AVX code implementation. Thanks, Paul Attachments: [application/octet-stream] v12-0001-Refactor-Split-pg_popcount-functions-into-multiple-f.patch (16.4K, ../../BL1PR11MB530478ACABE588A836338E74DC322@BL1PR11MB5304.namprd11.prod.outlook.com/2-v12-0001-Refactor-Split-pg_popcount-functions-into-multiple-f.patch) download | inline diff: From ba867ed01a1a25b2603eeda13a44e94a0a61648e Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Thu, 21 Mar 2024 11:19:23 -0700 Subject: [PATCH 1/2] [Refactor] Split pg_popcount functions into multiple files. Signed-off-by: Paul Amonson <[email protected]> --- src/include/port/pg_bitutils.h | 6 +- src/port/Makefile | 2 + src/port/meson.build | 2 + src/port/pg_bitutils.c | 232 +++------------------------ src/port/pg_popcount_x86_64_accel.c | 134 ++++++++++++++++ src/port/pg_popcount_x86_64_choose.c | 158 ++++++++++++++++++ 6 files changed, 324 insertions(+), 210 deletions(-) create mode 100644 src/port/pg_popcount_x86_64_accel.c create mode 100644 src/port/pg_popcount_x86_64_choose.c diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 53e5239717..26f6a48377 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -305,11 +305,13 @@ extern PGDLLIMPORT int (*pg_popcount64) (uint64 word); extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); #else -/* Use a portable implementation -- no need for a function pointer. */ +/* + * Use a portable implementation -- no need for a function pointer. Use + * inlining for small speed increase. + */ extern int pg_popcount32(uint32 word); extern int pg_popcount64(uint64 word); extern uint64 pg_popcount(const char *buf, int bytes); - #endif /* TRY_POPCNT_FAST */ /* diff --git a/src/port/Makefile b/src/port/Makefile index dcc8737e68..1499985dfc 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -44,6 +44,8 @@ OBJS = \ noblock.o \ path.o \ pg_bitutils.o \ + pg_popcount_x86_64_choose.o \ + pg_popcount_x86_64_accel.o \ pg_strong_random.o \ pgcheckdir.o \ pgmkdirp.o \ diff --git a/src/port/meson.build b/src/port/meson.build index 92b593e6ef..cf6e9fa06c 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -7,6 +7,8 @@ pgport_sources = [ 'noblock.c', 'path.c', 'pg_bitutils.c', + 'pg_popcount_x86_64_choose.c', + 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index 1197696e97..21a4d0ca97 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -21,7 +21,6 @@ #include "port/pg_bitutils.h" - /* * Array giving the position of the left-most set bit for each possible * byte value. We count the right-most position as the 0th bit, and the @@ -103,196 +102,46 @@ const uint8 pg_number_of_ones[256] = { 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8 }; -static inline int pg_popcount32_slow(uint32 word); -static inline int pg_popcount64_slow(uint64 word); -static uint64 pg_popcount_slow(const char *buf, int bytes); - -#ifdef TRY_POPCNT_FAST -static bool pg_popcount_available(void); -static int pg_popcount32_choose(uint32 word); -static int pg_popcount64_choose(uint64 word); -static uint64 pg_popcount_choose(const char *buf, int bytes); -static inline int pg_popcount32_fast(uint32 word); -static inline int pg_popcount64_fast(uint64 word); -static uint64 pg_popcount_fast(const char *buf, int bytes); - -int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; -int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; -uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; -#endif /* TRY_POPCNT_FAST */ - -#ifdef TRY_POPCNT_FAST - +#ifndef TRY_POPCNT_FAST /* - * Return true if CPUID indicates that the POPCNT instruction is available. + * Optimize function signature if using the slow functions. */ -static bool -pg_popcount_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; +#define INLINE static inline -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); #else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 23)) != 0; /* POPCNT */ -} - -/* - * These functions get called on the first call to pg_popcount32 etc. - * They detect whether we can use the asm implementations, and replace - * the function pointers so that subsequent calls are routed directly to - * the chosen implementation. - */ -static int -pg_popcount32_choose(uint32 word) -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - - return pg_popcount32(word); -} - -static int -pg_popcount64_choose(uint64 word) -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - - return pg_popcount64(word); -} - -static uint64 -pg_popcount_choose(const char *buf, int bytes) -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } +#define INLINE +#endif /* !TRY_POPCNT_FAST */ - return pg_popcount(buf, bytes); -} +/* Forward References */ +INLINE int pg_popcount32_slow(uint32 word); +INLINE int pg_popcount64_slow(uint64 word); +INLINE uint64 pg_popcount_slow(const char *buf, int bytes); -/* - * pg_popcount32_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount32_fast(uint32 word) +#ifndef TRY_POPCNT_FAST +/* Slow function defintions for exported functions. */ +int +pg_popcount32(uint32 word) { -#ifdef _MSC_VER - return __popcnt(word); -#else - uint32 res; - -__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif + return pg_popcount32_slow(word); } -/* - * pg_popcount64_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount64_fast(uint64 word) +int +pg_popcount64(uint64 word) { -#ifdef _MSC_VER - return __popcnt64(word); -#else - uint64 res; - -__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif + return pg_popcount64_slow(word); } - -/* - * pg_popcount_fast - * Returns the number of 1-bits in buf - */ -static uint64 -pg_popcount_fast(const char *buf, int bytes) +uint64 +pg_popcount(const char *buf, int bytes) { - uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) - { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_fast(*words++); - bytes -= 8; - } - - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_fast(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif - - /* Process any remaining bytes */ - while (bytes--) - popcnt += pg_number_of_ones[(unsigned char) *buf++]; - - return popcnt; + return pg_popcount_slow(buf, bytes); } - -#endif /* TRY_POPCNT_FAST */ - +#endif /* !TRY_POPCNT_FAST */ /* * pg_popcount32_slow * Return the number of 1 bits set in word */ -static inline int +INLINE int pg_popcount32_slow(uint32 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -314,7 +163,7 @@ pg_popcount32_slow(uint32 word) * pg_popcount64_slow * Return the number of 1 bits set in word */ -static inline int +INLINE int pg_popcount64_slow(uint64 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -342,7 +191,7 @@ pg_popcount64_slow(uint64 word) * pg_popcount_slow * Returns the number of 1-bits in buf */ -static uint64 +INLINE uint64 pg_popcount_slow(const char *buf, int bytes) { uint64 popcnt = 0; @@ -383,36 +232,3 @@ pg_popcount_slow(const char *buf, int bytes) return popcnt; } - -#ifndef TRY_POPCNT_FAST - -/* - * When the POPCNT instruction is not available, there's no point in using - * function pointers to vary the implementation between the fast and slow - * method. We instead just make these actual external functions when - * TRY_POPCNT_FAST is not defined. The compiler should be able to inline - * the slow versions here. - */ -int -pg_popcount32(uint32 word) -{ - return pg_popcount32_slow(word); -} - -int -pg_popcount64(uint64 word) -{ - return pg_popcount64_slow(word); -} - -/* - * pg_popcount - * Returns the number of 1-bits in buf - */ -uint64 -pg_popcount(const char *buf, int bytes) -{ - return pg_popcount_slow(buf, bytes); -} - -#endif /* !TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_accel.c b/src/port/pg_popcount_x86_64_accel.c new file mode 100644 index 0000000000..d5500d56e7 --- /dev/null +++ b/src/port/pg_popcount_x86_64_accel.c @@ -0,0 +1,134 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_accel.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2019-2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_accel.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" +#include "port/pg_bitutils.h" + +#if defined(HAVE__IMMINTRIN) +#include <immintrin.h> +#endif + +#ifdef TRY_POPCNT_FAST +int pg_popcount32_fast(uint32 word); +int pg_popcount64_fast(uint64 word); +uint64 pg_popcount_fast(const char *buf, int bytes); +uint64 pg_popcount512_fast(const char *buf, int bytes); + +/* + * pg_popcount32_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount32_fast(uint32 word) +{ +#ifdef _MSC_VER + return __popcnt(word); +#else + uint32 res; + +__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount64_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount64_fast(uint64 word) +{ +#ifdef _MSC_VER + return __popcnt64(word); +#else + uint64 res; + +__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount_fast + * Returns the number of 1-bits in buf + */ +uint64 +pg_popcount_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + +#if SIZEOF_VOID_P >= 8 + /* Process in 64-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(8, buf)) + { + const uint64 *words = (const uint64 *) buf; + + while (bytes >= 8) + { + popcnt += pg_popcount64_fast(*words++); + bytes -= 8; + } + + buf = (const char *) words; + } +#else + /* Process in 32-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(4, buf)) + { + const uint32 *words = (const uint32 *) buf; + + while (bytes >= 4) + { + popcnt += pg_popcount32_fast(*words++); + bytes -= 4; + } + + buf = (const char *) words; + } +#endif + + /* Process any remaining bytes */ + while (bytes--) + popcnt += pg_number_of_ones[(unsigned char) *buf++]; + + return popcnt; +} + +/* + * Use AVX-512 Intrinsics for supported CPUs or fall back the non-152 fast + * implem entation and use the best 64 bit fast methods. If no fast + * methods are used this will fall back to __builtin_* or pure software. + */ +uint64 +pg_popcount512_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + #if defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 + __m512i accumulator = _mm512_setzero_si512(); + + while (bytes >= 64) + { + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + + accumulator = _mm512_add_epi64(accumulator, p); + bytes -= 64; + buf += 64; + } + + popcnt = _mm512_reduce_add_epi64(accumulator); +#endif /* defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 */ + + /* Process any remaining bytes */ + return popcnt + pg_popcount_fast(buf, bytes); +} +#endif /* TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_choose.c b/src/port/pg_popcount_x86_64_choose.c new file mode 100644 index 0000000000..e73d1999ad --- /dev/null +++ b/src/port/pg_popcount_x86_64_choose.c @@ -0,0 +1,158 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_choose.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2019-2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_choose.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" +#include "port/pg_bitutils.h" + +#ifdef TRY_POPCNT_FAST + +#ifdef HAVE__GET_CPUID +#include <cpuid.h> +#endif +#ifdef HAVE__CPUID +#include <intrin.h> +#endif + +static bool pg_popcount_available(void); +static int pg_popcount32_choose(uint32 word); +static int pg_popcount64_choose(uint64 word); +static uint64 pg_popcount_choose(const char *buf, int bytes); +extern int pg_popcount32_fast(uint32 word); +extern int pg_popcount64_fast(uint64 word); +extern uint64 pg_popcount_fast(const char *buf, int bytes); +extern uint64 pg_popcount512_fast(const char *buf, int bytes); +extern int pg_popcount32_slow(uint32 word); +extern int pg_popcount64_slow(uint64 word); +extern uint64 pg_popcount_slow(const char *buf, int bytes); + +int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; +int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; +uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; + +/* + * Return true if CPUID indicates that the POPCNT instruction is available. + */ +static bool +pg_popcount_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + +#if defined(HAVE__GET_CPUID) + __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUID) + __cpuid(exx, 1); +#else +#error cpuid instruction not available +#endif + + return (exx[2] & (1 << 23)) != 0; /* POPCNT */ +} + +/* + * Return true if CPUID indicates that the AVX512_POPCNT instruction is + * available. This is similar to the method above; see + * https://en.wikipedia.org/wiki/CPUID#EAX=7,_ECX=0:_Extended_Features + * + * Finally, we make sure the xgetbv result is consistent with the CPUID + * results. + */ +static bool +pg_popcount512_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + + /* Check for AVX512VPOPCNTDQ and AVX512F */ +#if defined(HAVE__GET_CPUID_COUNT) + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUIDEX) + __cpuidex(exx, 7, 0); +#endif + + if ((exx[2] & (0x00004000)) != 0 && (exx[1] & (0x00010000)) != 0) + { + /* + * CPUID succeeded, does the current running OS support the + * ZMM registers which are required for AVX512? This check is + * required to make sure an old OS on a new CPU is correctly + * checked or a VM hypervisor is not excluding AVX512 ZMM + * support in the VM; see "5.1.9 Detection of AVX Instructions" + * https://www.intel.com/content/www/us/en/content-details/671488/intel-64-and-ia-32-architectures-optimization-reference-manual-volume-1.html + */ + uint64 xcr = 0; +#ifdef _MSC_VER + uint64 highlow = _xgetbv(xcr); + + return (highlow & 0xE0) != 0; +#else + uint32 high; + uint32 low; + + __asm__ __volatile__("xgetbv\t\n" : "=a"(low), "=d"(high) : "c"(xcr)); + return (low & 0xE0) != 0; +#endif + } /* POPCNT 512 */ + return false; +} + +/* + * These functions get called on the first call to pg_popcount32 etc. + * They detect whether we can use the asm implementations, and replace + * the function pointers so that subsequent calls are routed directly to + * the chosen implementation. + */ +static inline void set_function_pointers() +{ +if (pg_popcount512_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + if (pg_popcount_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + pg_popcount32 = pg_popcount32_slow; + pg_popcount64 = pg_popcount64_slow; + pg_popcount = pg_popcount_slow; + } + } +} + +static inline int +pg_popcount32_choose(uint32 word) +{ + set_function_pointers(); + return pg_popcount32(word); +} + +static inline int +pg_popcount64_choose(uint64 word) +{ + set_function_pointers(); + return pg_popcount64(word); +} + +static inline uint64 +pg_popcount_choose(const char *buf, int bytes) +{ + set_function_pointers(); + return pg_popcount(buf, bytes); +} + +#endif /* TRY_POPCNT_FAST */ -- 2.34.1 [application/octet-stream] v12-0002-Feature-Added-AVX-512-acceleration-to-the-pg_popcoun.patch (17.9K, ../../BL1PR11MB530478ACABE588A836338E74DC322@BL1PR11MB5304.namprd11.prod.outlook.com/3-v12-0002-Feature-Added-AVX-512-acceleration-to-the-pg_popcoun.patch) download | inline diff: From 86415641d953ef7bc36fe91f75ce9c69878cba63 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Thu, 21 Mar 2024 11:39:37 -0700 Subject: [PATCH 2/2] [Feature] Added AVX-512 acceleration to the pg_popcount function. Signed-off-by: Paul Amonson <[email protected]> --- config/c-compiler.m4 | 37 ++++++ configure | 205 +++++++++++++++++++++++++++++++++ configure.ac | 44 +++++++ meson.build | 72 ++++++++++++ src/Makefile.global.in | 1 + src/include/pg_config.h.in | 12 ++ src/include/port/pg_bitutils.h | 4 +- src/makefiles/meson.build | 1 + src/port/Makefile | 5 + src/port/meson.build | 6 +- 10 files changed, 382 insertions(+), 5 deletions(-) diff --git a/config/c-compiler.m4 b/config/c-compiler.m4 index 3268a780bb..94e3e713aa 100644 --- a/config/c-compiler.m4 +++ b/config/c-compiler.m4 @@ -694,3 +694,40 @@ if test x"$Ac_cachevar" = x"yes"; then fi undefine([Ac_cachevar])dnl ])# PGAC_LOONGARCH_CRC32C_INTRINSICS + +# PGAC_AVX512_POPCNT_INTRINSICS +# --------------------------- +# Check if the compiler supports the x86_64 AVX512 POPCNT instructions using +# intrinsics used in CPUID features AVX512F and AVX512VPOPCNTDQ. +# +# Optional compiler flags can be passed as argument (e.g. -mavx512vpopcntdq). +# If the intrinsics are supported then pgac_avx512_popcnt_intrinsics and +# CFLAGS_AVX512_POPCNT are set. +AC_DEFUN([PGAC_AVX512_POPCNT_INTRINSICS], +[define([Ac_cachevar], [AS_TR_SH([pgac_cv_avx512_popcnt_intrinsics_$1])])dnl +AC_CACHE_CHECK([for _mm512_popcnt_epi64 with CFLAGS=$1], [Ac_cachevar], +[pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS $1" +AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h>], + [const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0;])], + [Ac_cachevar=yes], + [Ac_cachevar=no]) +CFLAGS="$pgac_save_CFLAGS"]) +if test x"$Ac_cachevar" = x"yes"; then + CFLAGS_AVX512_POPCNT="$1" + pgac_avx512_popcnt_intrinsics=yes +fi +undefine([Ac_cachevar])dnl +])# PGAC_AVX512_POPCNT_INTRINSICS diff --git a/configure b/configure index 36feeafbb2..7253cb8154 100755 --- a/configure +++ b/configure @@ -647,6 +647,7 @@ MSGFMT_FLAGS MSGFMT PG_CRC32C_OBJS CFLAGS_CRC +CFLAGS_AVX512_POPCNT LIBOBJS OPENSSL ZSTD @@ -17404,6 +17405,41 @@ $as_echo "#define HAVE__GET_CPUID 1" >>confdefs.h fi +# Check for x86 cpuid_count instruction +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __get_cpuid_count" >&5 +$as_echo_n "checking for __get_cpuid_count... " >&6; } +if ${pgac_cv__get_cpuid_count+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <cpuid.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__get_cpuid_count="yes" +else + pgac_cv__get_cpuid_count="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__get_cpuid_count" >&5 +$as_echo "$pgac_cv__get_cpuid_count" >&6; } +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + +$as_echo "#define HAVE__GET_CPUID_COUNT 1" >>confdefs.h + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuid" >&5 $as_echo_n "checking for __cpuid... " >&6; } if ${pgac_cv__cpuid+:} false; then : @@ -17438,6 +17474,175 @@ $as_echo "#define HAVE__CPUID 1" >>confdefs.h fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuidex" >&5 +$as_echo_n "checking for __cpuidex... " >&6; } +if ${pgac_cv__cpuidex+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <intrin.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__cpuidex="yes" +else + pgac_cv__cpuidex="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__cpuidex" >&5 +$as_echo "$pgac_cv__cpuidex" >&6; } +if test x"$pgac_cv__cpuidex" = x"yes"; then + +$as_echo "#define HAVE__CPUIDEX 1" >>confdefs.h + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __immintrin" >&5 +$as_echo_n "checking for __immintrin... " >&6; } +if ${pgac_cv__immintrin+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +int +main () +{ +/* Don't exclude code so added return. */ + return 1701; + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__immintrin="yes" +else + pgac_cv__immintrin="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__immintrin" >&5 +$as_echo "$pgac_cv__immintrin" >&6; } +if test x"$pgac_cv__immintrin" = x"yes"; then + +$as_echo "#define HAVE__IMMINTRIN 1" >>confdefs.h + +fi + +# Check for AVX512 intrinsics to do POPCNT calculations. +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics_+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS " +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics_=yes +else + pgac_cv_avx512_popcnt_intrinsics_=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics_" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics_" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics_" = x"yes"; then + CFLAGS_AVX512_POPCNT="" + pgac_avx512_popcnt_intrinsics=yes +fi + +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS -mavx512vpopcntdq" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=yes +else + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" = x"yes"; then + CFLAGS_AVX512_POPCNT="-mavx512vpopcntdq" + pgac_avx512_popcnt_intrinsics=yes +fi + + +$as_echo "#define HAVE__AVX512_POPCNT 1" >>confdefs.h + +fi + + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/configure.ac b/configure.ac index 57f734879e..06e8eaa6d0 100644 --- a/configure.ac +++ b/configure.ac @@ -2052,6 +2052,18 @@ if test x"$pgac_cv__get_cpuid" = x"yes"; then AC_DEFINE(HAVE__GET_CPUID, 1, [Define to 1 if you have __get_cpuid.]) fi +# Check for x86 cpuid_count instruction +AC_CACHE_CHECK([for __get_cpuid_count], [pgac_cv__get_cpuid_count], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <cpuid.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + ]])], + [pgac_cv__get_cpuid_count="yes"], + [pgac_cv__get_cpuid_count="no"])]) +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + AC_DEFINE(HAVE__GET_CPUID_COUNT, 1, [Define to 1 if you have __get_cpuid.]) +fi + AC_CACHE_CHECK([for __cpuid], [pgac_cv__cpuid], [AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], [[unsigned int exx[4] = {0, 0, 0, 0}; @@ -2063,6 +2075,38 @@ if test x"$pgac_cv__cpuid" = x"yes"; then AC_DEFINE(HAVE__CPUID, 1, [Define to 1 if you have __cpuid.]) fi +AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + ]])], + [pgac_cv__cpuidex="yes"], + [pgac_cv__cpuidex="no"])]) +if test x"$pgac_cv__cpuidex" = x"yes"; then + AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.]) +fi + +AC_CACHE_CHECK([for __immintrin], [pgac_cv__immintrin], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h>], + [[/* Don't exclude code so added return. */ + return 1701; + ]])], + [pgac_cv__immintrin="yes"], + [pgac_cv__immintrin="no"])]) +if test x"$pgac_cv__immintrin" = x"yes"; then + AC_DEFINE(HAVE__IMMINTRIN, 1, [Define to 1 if you have immintrin.]) +fi + +# Check for AVX512 intrinsics to do POPCNT calculations. +# +PGAC_AVX512_POPCNT_INTRINSICS([]) +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + PGAC_AVX512_POPCNT_INTRINSICS([-mavx512vpopcntdq]) + AC_DEFINE(HAVE__AVX512_POPCNT, 1, [Define to 1 if you have cpu + support for AVX512 POPCNT.]) +fi +AC_SUBST(CFLAGS_AVX512_POPCNT) + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/meson.build b/meson.build index c8fdfeb0ec..cd508096e5 100644 --- a/meson.build +++ b/meson.build @@ -1783,6 +1783,37 @@ elif cc.links(''' endif +if cc.links(''' + #include <cpuid.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + } + ''', name: '__get_cpuid_count', + args: test_c_args) + cdata.set('HAVE__GET_CPUID_COUNT', 1) +elif cc.links(''' + #include <intrin.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __cpuidex(exx, 7, 0); + } + ''', name: '__cpuidex', + args: test_c_args) + cdata.set('HAVE__CPUIDEX', 1) +endif + + +# Check for header immintrin.h +if cc.has_header('immintrin.h', + include_directories: postgres_inc, args: test_c_args) + cdata.set('HAVE__IMMINTRIN', 1, + description: 'Define to 1 if you have the immintrin.h header file.') +endif + + # Defend against clang being used on x86-32 without SSE2 enabled. As current # versions of clang do not understand -fexcess-precision=standard, the use of # x87 floating point operations leads to problems like isinf possibly returning @@ -2158,6 +2189,47 @@ endif +############################################################### +# AVX 512 POPCNT Intrinsic check +############################################################### +have_avx512_popcnt = false +cflags_avx512_popcnt = [] +if host_cpu == 'x86_64' + test_flags = ['-mavx512vpopcntdq'] + if host_system == 'windows' + test_flags = ['/arch:AVX512'] + endif + prog = ''' + #include <immintrin.h> + #include <stdint.h> + #include <stdlib.h> + #include <string.h> + void main(void) + { + const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + }''' + if cc.links(prog, name: '_mm512_* methods with -mavx512vpopcntdq flag.', + args: test_c_args + test_flags) + have_avx512_popcnt = true + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cdata.set('HAVE__AVX512_POPCNT', 1) + cflags_avx512_popcnt = test_flags + else + have_avx512_popcnt = false + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cflags_avx512_popcnt = [] + endif # compile/link test +endif # host_cpu check + ############################################################### # Library / OS tests ############################################################### diff --git a/src/Makefile.global.in b/src/Makefile.global.in index 8b3f8c24e0..089f49b7f3 100644 --- a/src/Makefile.global.in +++ b/src/Makefile.global.in @@ -263,6 +263,7 @@ CXXFLAGS_SL_MODULE = @CXXFLAGS_SL_MODULE@ CFLAGS_UNROLL_LOOPS = @CFLAGS_UNROLL_LOOPS@ CFLAGS_VECTORIZE = @CFLAGS_VECTORIZE@ CFLAGS_CRC = @CFLAGS_CRC@ +CFLAGS_AVX512_POPCNT = @CFLAGS_AVX512_POPCNT@ PERMIT_DECLARATION_AFTER_STATEMENT = @PERMIT_DECLARATION_AFTER_STATEMENT@ CXXFLAGS = @CXXFLAGS@ diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 591e1ca3df..33a831e768 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -558,6 +558,18 @@ /* Define to 1 if you have __get_cpuid. */ #undef HAVE__GET_CPUID +/* Define to 1 if you have __get_cpuid_count. */ +#undef HAVE__GET_CPUID_COUNT + +/* Define to 1 if you have __get_cpuidex. */ +#undef HAVE__GET_CPUIDEX + +/* Define to 1 if you have immintrin. */ +#undef HAVE__IMMINTRIN + +/* Define to 1 if you have AVX512. */ +#undef HAVE__AVX512_POPCNT + /* Define to 1 if your compiler understands _Static_assert. */ #undef HAVE__STATIC_ASSERT diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 26f6a48377..8215316b0e 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -306,8 +306,8 @@ extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); #else /* - * Use a portable implementation -- no need for a function pointer. Use - * inlining for small speed increase. + * Use a portable implementation -- no need for a function pointer. Use + * inlining for small speed increase. */ extern int pg_popcount32(uint32 word); extern int pg_popcount64(uint64 word); diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build index b0f4178b3d..ee3647282e 100644 --- a/src/makefiles/meson.build +++ b/src/makefiles/meson.build @@ -100,6 +100,7 @@ pgxs_kv = { ' '.join(cflags_no_decl_after_statement), 'CFLAGS_CRC': ' '.join(cflags_crc), + 'CFLAGS_AVX512_POPCNT': ' '.join(cflags_avx512_popcnt), 'CFLAGS_UNROLL_LOOPS': ' '.join(unroll_loops_cflags), 'CFLAGS_VECTORIZE': ' '.join(vectorize_cflags), diff --git a/src/port/Makefile b/src/port/Makefile index 1499985dfc..92bd419953 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -89,6 +89,11 @@ pg_crc32c_sse42.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_shlib.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_srv.o: CFLAGS+=$(CFLAGS_CRC) +# Newer processors can use AVX-512 POPCNT Capabilities +pg_popcount_x86_64_accel.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_shlib.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_srv.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) + # all versions of pg_crc32c_armv8.o need CFLAGS_CRC pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC) diff --git a/src/port/meson.build b/src/port/meson.build index cf6e9fa06c..0647e7a4f7 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,7 +8,6 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_x86_64_choose.c', - 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', @@ -86,6 +85,7 @@ replace_funcs_pos = [ ['pg_crc32c_sse42', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK', 'crc'], ['pg_crc32c_sse42_choose', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], ['pg_crc32c_sb8', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], + ['pg_popcount_x86_64_accel', 'USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 'avx512'], # arm / aarch64 ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C'], @@ -100,8 +100,8 @@ replace_funcs_pos = [ ['pg_crc32c_sb8', 'USE_SLICING_BY_8_CRC32C'], ] -pgport_cflags = {'crc': cflags_crc} -pgport_sources_cflags = {'crc': []} +pgport_cflags = {'crc': cflags_crc, 'avx512': cflags_avx512_popcnt} +pgport_sources_cflags = {'crc': [], 'avx512': []} foreach f : replace_funcs_neg func = f.get(0) -- 2.34.1 ^ permalink raw reply [nested|flat] 12+ messages in thread
* RE: Popcount optimization using AVX512 @ 2024-03-25 15:06 Amonson, Paul D <[email protected]> parent: Amonson, Paul D <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Amonson, Paul D @ 2024-03-25 15:06 UTC (permalink / raw) To: Amonson, Paul D <[email protected]>; David Rowley <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Tom Lane <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> > -----Original Message----- > From: Amonson, Paul D <[email protected]> > Sent: Thursday, March 21, 2024 12:18 PM > To: David Rowley <[email protected]> > Cc: Nathan Bossart <[email protected]>; Andres Freund I am re-posting the patches as CI for Mac failed (CI error not code/test error). The patches are the same as last time. Thanks, Paul Attachments: [application/octet-stream] v12-0001-Refactor-Split-pg_popcount-functions-into-multiple-f.patch (16.4K, ../../BL1PR11MB53040D8E18518B2BDA4BB83ADC362@BL1PR11MB5304.namprd11.prod.outlook.com/2-v12-0001-Refactor-Split-pg_popcount-functions-into-multiple-f.patch) download | inline diff: From ba867ed01a1a25b2603eeda13a44e94a0a61648e Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Thu, 21 Mar 2024 11:19:23 -0700 Subject: [PATCH 1/2] [Refactor] Split pg_popcount functions into multiple files. Signed-off-by: Paul Amonson <[email protected]> --- src/include/port/pg_bitutils.h | 6 +- src/port/Makefile | 2 + src/port/meson.build | 2 + src/port/pg_bitutils.c | 232 +++------------------------ src/port/pg_popcount_x86_64_accel.c | 134 ++++++++++++++++ src/port/pg_popcount_x86_64_choose.c | 158 ++++++++++++++++++ 6 files changed, 324 insertions(+), 210 deletions(-) create mode 100644 src/port/pg_popcount_x86_64_accel.c create mode 100644 src/port/pg_popcount_x86_64_choose.c diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 53e5239717..26f6a48377 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -305,11 +305,13 @@ extern PGDLLIMPORT int (*pg_popcount64) (uint64 word); extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); #else -/* Use a portable implementation -- no need for a function pointer. */ +/* + * Use a portable implementation -- no need for a function pointer. Use + * inlining for small speed increase. + */ extern int pg_popcount32(uint32 word); extern int pg_popcount64(uint64 word); extern uint64 pg_popcount(const char *buf, int bytes); - #endif /* TRY_POPCNT_FAST */ /* diff --git a/src/port/Makefile b/src/port/Makefile index dcc8737e68..1499985dfc 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -44,6 +44,8 @@ OBJS = \ noblock.o \ path.o \ pg_bitutils.o \ + pg_popcount_x86_64_choose.o \ + pg_popcount_x86_64_accel.o \ pg_strong_random.o \ pgcheckdir.o \ pgmkdirp.o \ diff --git a/src/port/meson.build b/src/port/meson.build index 92b593e6ef..cf6e9fa06c 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -7,6 +7,8 @@ pgport_sources = [ 'noblock.c', 'path.c', 'pg_bitutils.c', + 'pg_popcount_x86_64_choose.c', + 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index 1197696e97..21a4d0ca97 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -21,7 +21,6 @@ #include "port/pg_bitutils.h" - /* * Array giving the position of the left-most set bit for each possible * byte value. We count the right-most position as the 0th bit, and the @@ -103,196 +102,46 @@ const uint8 pg_number_of_ones[256] = { 4, 5, 5, 6, 5, 6, 6, 7, 5, 6, 6, 7, 6, 7, 7, 8 }; -static inline int pg_popcount32_slow(uint32 word); -static inline int pg_popcount64_slow(uint64 word); -static uint64 pg_popcount_slow(const char *buf, int bytes); - -#ifdef TRY_POPCNT_FAST -static bool pg_popcount_available(void); -static int pg_popcount32_choose(uint32 word); -static int pg_popcount64_choose(uint64 word); -static uint64 pg_popcount_choose(const char *buf, int bytes); -static inline int pg_popcount32_fast(uint32 word); -static inline int pg_popcount64_fast(uint64 word); -static uint64 pg_popcount_fast(const char *buf, int bytes); - -int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; -int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; -uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; -#endif /* TRY_POPCNT_FAST */ - -#ifdef TRY_POPCNT_FAST - +#ifndef TRY_POPCNT_FAST /* - * Return true if CPUID indicates that the POPCNT instruction is available. + * Optimize function signature if using the slow functions. */ -static bool -pg_popcount_available(void) -{ - unsigned int exx[4] = {0, 0, 0, 0}; +#define INLINE static inline -#if defined(HAVE__GET_CPUID) - __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); -#elif defined(HAVE__CPUID) - __cpuid(exx, 1); #else -#error cpuid instruction not available -#endif - - return (exx[2] & (1 << 23)) != 0; /* POPCNT */ -} - -/* - * These functions get called on the first call to pg_popcount32 etc. - * They detect whether we can use the asm implementations, and replace - * the function pointers so that subsequent calls are routed directly to - * the chosen implementation. - */ -static int -pg_popcount32_choose(uint32 word) -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - - return pg_popcount32(word); -} - -static int -pg_popcount64_choose(uint64 word) -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } - - return pg_popcount64(word); -} - -static uint64 -pg_popcount_choose(const char *buf, int bytes) -{ - if (pg_popcount_available()) - { - pg_popcount32 = pg_popcount32_fast; - pg_popcount64 = pg_popcount64_fast; - pg_popcount = pg_popcount_fast; - } - else - { - pg_popcount32 = pg_popcount32_slow; - pg_popcount64 = pg_popcount64_slow; - pg_popcount = pg_popcount_slow; - } +#define INLINE +#endif /* !TRY_POPCNT_FAST */ - return pg_popcount(buf, bytes); -} +/* Forward References */ +INLINE int pg_popcount32_slow(uint32 word); +INLINE int pg_popcount64_slow(uint64 word); +INLINE uint64 pg_popcount_slow(const char *buf, int bytes); -/* - * pg_popcount32_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount32_fast(uint32 word) +#ifndef TRY_POPCNT_FAST +/* Slow function defintions for exported functions. */ +int +pg_popcount32(uint32 word) { -#ifdef _MSC_VER - return __popcnt(word); -#else - uint32 res; - -__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif + return pg_popcount32_slow(word); } -/* - * pg_popcount64_fast - * Return the number of 1 bits set in word - */ -static inline int -pg_popcount64_fast(uint64 word) +int +pg_popcount64(uint64 word) { -#ifdef _MSC_VER - return __popcnt64(word); -#else - uint64 res; - -__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); - return (int) res; -#endif + return pg_popcount64_slow(word); } - -/* - * pg_popcount_fast - * Returns the number of 1-bits in buf - */ -static uint64 -pg_popcount_fast(const char *buf, int bytes) +uint64 +pg_popcount(const char *buf, int bytes) { - uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) - { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_fast(*words++); - bytes -= 8; - } - - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_fast(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif - - /* Process any remaining bytes */ - while (bytes--) - popcnt += pg_number_of_ones[(unsigned char) *buf++]; - - return popcnt; + return pg_popcount_slow(buf, bytes); } - -#endif /* TRY_POPCNT_FAST */ - +#endif /* !TRY_POPCNT_FAST */ /* * pg_popcount32_slow * Return the number of 1 bits set in word */ -static inline int +INLINE int pg_popcount32_slow(uint32 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -314,7 +163,7 @@ pg_popcount32_slow(uint32 word) * pg_popcount64_slow * Return the number of 1 bits set in word */ -static inline int +INLINE int pg_popcount64_slow(uint64 word) { #ifdef HAVE__BUILTIN_POPCOUNT @@ -342,7 +191,7 @@ pg_popcount64_slow(uint64 word) * pg_popcount_slow * Returns the number of 1-bits in buf */ -static uint64 +INLINE uint64 pg_popcount_slow(const char *buf, int bytes) { uint64 popcnt = 0; @@ -383,36 +232,3 @@ pg_popcount_slow(const char *buf, int bytes) return popcnt; } - -#ifndef TRY_POPCNT_FAST - -/* - * When the POPCNT instruction is not available, there's no point in using - * function pointers to vary the implementation between the fast and slow - * method. We instead just make these actual external functions when - * TRY_POPCNT_FAST is not defined. The compiler should be able to inline - * the slow versions here. - */ -int -pg_popcount32(uint32 word) -{ - return pg_popcount32_slow(word); -} - -int -pg_popcount64(uint64 word) -{ - return pg_popcount64_slow(word); -} - -/* - * pg_popcount - * Returns the number of 1-bits in buf - */ -uint64 -pg_popcount(const char *buf, int bytes) -{ - return pg_popcount_slow(buf, bytes); -} - -#endif /* !TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_accel.c b/src/port/pg_popcount_x86_64_accel.c new file mode 100644 index 0000000000..d5500d56e7 --- /dev/null +++ b/src/port/pg_popcount_x86_64_accel.c @@ -0,0 +1,134 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_accel.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2019-2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_accel.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" +#include "port/pg_bitutils.h" + +#if defined(HAVE__IMMINTRIN) +#include <immintrin.h> +#endif + +#ifdef TRY_POPCNT_FAST +int pg_popcount32_fast(uint32 word); +int pg_popcount64_fast(uint64 word); +uint64 pg_popcount_fast(const char *buf, int bytes); +uint64 pg_popcount512_fast(const char *buf, int bytes); + +/* + * pg_popcount32_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount32_fast(uint32 word) +{ +#ifdef _MSC_VER + return __popcnt(word); +#else + uint32 res; + +__asm__ __volatile__(" popcntl %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount64_fast + * Return the number of 1 bits set in word + */ +int +pg_popcount64_fast(uint64 word) +{ +#ifdef _MSC_VER + return __popcnt64(word); +#else + uint64 res; + +__asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); + return (int) res; +#endif +} + +/* + * pg_popcount_fast + * Returns the number of 1-bits in buf + */ +uint64 +pg_popcount_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + +#if SIZEOF_VOID_P >= 8 + /* Process in 64-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(8, buf)) + { + const uint64 *words = (const uint64 *) buf; + + while (bytes >= 8) + { + popcnt += pg_popcount64_fast(*words++); + bytes -= 8; + } + + buf = (const char *) words; + } +#else + /* Process in 32-bit chunks if the buffer is aligned. */ + if (buf == (const char *) TYPEALIGN(4, buf)) + { + const uint32 *words = (const uint32 *) buf; + + while (bytes >= 4) + { + popcnt += pg_popcount32_fast(*words++); + bytes -= 4; + } + + buf = (const char *) words; + } +#endif + + /* Process any remaining bytes */ + while (bytes--) + popcnt += pg_number_of_ones[(unsigned char) *buf++]; + + return popcnt; +} + +/* + * Use AVX-512 Intrinsics for supported CPUs or fall back the non-152 fast + * implem entation and use the best 64 bit fast methods. If no fast + * methods are used this will fall back to __builtin_* or pure software. + */ +uint64 +pg_popcount512_fast(const char *buf, int bytes) +{ + uint64 popcnt = 0; + #if defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 + __m512i accumulator = _mm512_setzero_si512(); + + while (bytes >= 64) + { + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + + accumulator = _mm512_add_epi64(accumulator, p); + bytes -= 64; + buf += 64; + } + + popcnt = _mm512_reduce_add_epi64(accumulator); +#endif /* defined(HAVE__IMMINTRIN) && HAVE__AVX512_POPCNT == 1 */ + + /* Process any remaining bytes */ + return popcnt + pg_popcount_fast(buf, bytes); +} +#endif /* TRY_POPCNT_FAST */ diff --git a/src/port/pg_popcount_x86_64_choose.c b/src/port/pg_popcount_x86_64_choose.c new file mode 100644 index 0000000000..e73d1999ad --- /dev/null +++ b/src/port/pg_popcount_x86_64_choose.c @@ -0,0 +1,158 @@ +/*------------------------------------------------------------------------- + * + * pg_popcount_x86_64_choose.c + * Miscellaneous functions for bit-wise operations. + * + * Copyright (c) 2019-2024, PostgreSQL Global Development Group + * + * IDENTIFICATION + * src/port/pg_popcount_x86_64_choose.c + * + *------------------------------------------------------------------------- + */ +#include "c.h" +#include "port/pg_bitutils.h" + +#ifdef TRY_POPCNT_FAST + +#ifdef HAVE__GET_CPUID +#include <cpuid.h> +#endif +#ifdef HAVE__CPUID +#include <intrin.h> +#endif + +static bool pg_popcount_available(void); +static int pg_popcount32_choose(uint32 word); +static int pg_popcount64_choose(uint64 word); +static uint64 pg_popcount_choose(const char *buf, int bytes); +extern int pg_popcount32_fast(uint32 word); +extern int pg_popcount64_fast(uint64 word); +extern uint64 pg_popcount_fast(const char *buf, int bytes); +extern uint64 pg_popcount512_fast(const char *buf, int bytes); +extern int pg_popcount32_slow(uint32 word); +extern int pg_popcount64_slow(uint64 word); +extern uint64 pg_popcount_slow(const char *buf, int bytes); + +int (*pg_popcount32) (uint32 word) = pg_popcount32_choose; +int (*pg_popcount64) (uint64 word) = pg_popcount64_choose; +uint64 (*pg_popcount) (const char *buf, int bytes) = pg_popcount_choose; + +/* + * Return true if CPUID indicates that the POPCNT instruction is available. + */ +static bool +pg_popcount_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + +#if defined(HAVE__GET_CPUID) + __get_cpuid(1, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUID) + __cpuid(exx, 1); +#else +#error cpuid instruction not available +#endif + + return (exx[2] & (1 << 23)) != 0; /* POPCNT */ +} + +/* + * Return true if CPUID indicates that the AVX512_POPCNT instruction is + * available. This is similar to the method above; see + * https://en.wikipedia.org/wiki/CPUID#EAX=7,_ECX=0:_Extended_Features + * + * Finally, we make sure the xgetbv result is consistent with the CPUID + * results. + */ +static bool +pg_popcount512_available(void) +{ + unsigned int exx[4] = {0, 0, 0, 0}; + + /* Check for AVX512VPOPCNTDQ and AVX512F */ +#if defined(HAVE__GET_CPUID_COUNT) + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); +#elif defined(HAVE__CPUIDEX) + __cpuidex(exx, 7, 0); +#endif + + if ((exx[2] & (0x00004000)) != 0 && (exx[1] & (0x00010000)) != 0) + { + /* + * CPUID succeeded, does the current running OS support the + * ZMM registers which are required for AVX512? This check is + * required to make sure an old OS on a new CPU is correctly + * checked or a VM hypervisor is not excluding AVX512 ZMM + * support in the VM; see "5.1.9 Detection of AVX Instructions" + * https://www.intel.com/content/www/us/en/content-details/671488/intel-64-and-ia-32-architectures-optimization-reference-manual-volume-1.html + */ + uint64 xcr = 0; +#ifdef _MSC_VER + uint64 highlow = _xgetbv(xcr); + + return (highlow & 0xE0) != 0; +#else + uint32 high; + uint32 low; + + __asm__ __volatile__("xgetbv\t\n" : "=a"(low), "=d"(high) : "c"(xcr)); + return (low & 0xE0) != 0; +#endif + } /* POPCNT 512 */ + return false; +} + +/* + * These functions get called on the first call to pg_popcount32 etc. + * They detect whether we can use the asm implementations, and replace + * the function pointers so that subsequent calls are routed directly to + * the chosen implementation. + */ +static inline void set_function_pointers() +{ +if (pg_popcount512_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + if (pg_popcount_available()) + { + pg_popcount32 = pg_popcount32_fast; + pg_popcount64 = pg_popcount64_fast; + pg_popcount = pg_popcount_fast; + } + else + { + pg_popcount32 = pg_popcount32_slow; + pg_popcount64 = pg_popcount64_slow; + pg_popcount = pg_popcount_slow; + } + } +} + +static inline int +pg_popcount32_choose(uint32 word) +{ + set_function_pointers(); + return pg_popcount32(word); +} + +static inline int +pg_popcount64_choose(uint64 word) +{ + set_function_pointers(); + return pg_popcount64(word); +} + +static inline uint64 +pg_popcount_choose(const char *buf, int bytes) +{ + set_function_pointers(); + return pg_popcount(buf, bytes); +} + +#endif /* TRY_POPCNT_FAST */ -- 2.34.1 [application/octet-stream] v12-0002-Feature-Added-AVX-512-acceleration-to-the-pg_popcoun.patch (17.9K, ../../BL1PR11MB53040D8E18518B2BDA4BB83ADC362@BL1PR11MB5304.namprd11.prod.outlook.com/3-v12-0002-Feature-Added-AVX-512-acceleration-to-the-pg_popcoun.patch) download | inline diff: From 86415641d953ef7bc36fe91f75ce9c69878cba63 Mon Sep 17 00:00:00 2001 From: Paul Amonson <[email protected]> Date: Thu, 21 Mar 2024 11:39:37 -0700 Subject: [PATCH 2/2] [Feature] Added AVX-512 acceleration to the pg_popcount function. Signed-off-by: Paul Amonson <[email protected]> --- config/c-compiler.m4 | 37 ++++++ configure | 205 +++++++++++++++++++++++++++++++++ configure.ac | 44 +++++++ meson.build | 72 ++++++++++++ src/Makefile.global.in | 1 + src/include/pg_config.h.in | 12 ++ src/include/port/pg_bitutils.h | 4 +- src/makefiles/meson.build | 1 + src/port/Makefile | 5 + src/port/meson.build | 6 +- 10 files changed, 382 insertions(+), 5 deletions(-) diff --git a/config/c-compiler.m4 b/config/c-compiler.m4 index 3268a780bb..94e3e713aa 100644 --- a/config/c-compiler.m4 +++ b/config/c-compiler.m4 @@ -694,3 +694,40 @@ if test x"$Ac_cachevar" = x"yes"; then fi undefine([Ac_cachevar])dnl ])# PGAC_LOONGARCH_CRC32C_INTRINSICS + +# PGAC_AVX512_POPCNT_INTRINSICS +# --------------------------- +# Check if the compiler supports the x86_64 AVX512 POPCNT instructions using +# intrinsics used in CPUID features AVX512F and AVX512VPOPCNTDQ. +# +# Optional compiler flags can be passed as argument (e.g. -mavx512vpopcntdq). +# If the intrinsics are supported then pgac_avx512_popcnt_intrinsics and +# CFLAGS_AVX512_POPCNT are set. +AC_DEFUN([PGAC_AVX512_POPCNT_INTRINSICS], +[define([Ac_cachevar], [AS_TR_SH([pgac_cv_avx512_popcnt_intrinsics_$1])])dnl +AC_CACHE_CHECK([for _mm512_popcnt_epi64 with CFLAGS=$1], [Ac_cachevar], +[pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS $1" +AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h>], + [const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0;])], + [Ac_cachevar=yes], + [Ac_cachevar=no]) +CFLAGS="$pgac_save_CFLAGS"]) +if test x"$Ac_cachevar" = x"yes"; then + CFLAGS_AVX512_POPCNT="$1" + pgac_avx512_popcnt_intrinsics=yes +fi +undefine([Ac_cachevar])dnl +])# PGAC_AVX512_POPCNT_INTRINSICS diff --git a/configure b/configure index 36feeafbb2..7253cb8154 100755 --- a/configure +++ b/configure @@ -647,6 +647,7 @@ MSGFMT_FLAGS MSGFMT PG_CRC32C_OBJS CFLAGS_CRC +CFLAGS_AVX512_POPCNT LIBOBJS OPENSSL ZSTD @@ -17404,6 +17405,41 @@ $as_echo "#define HAVE__GET_CPUID 1" >>confdefs.h fi +# Check for x86 cpuid_count instruction +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __get_cpuid_count" >&5 +$as_echo_n "checking for __get_cpuid_count... " >&6; } +if ${pgac_cv__get_cpuid_count+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <cpuid.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__get_cpuid_count="yes" +else + pgac_cv__get_cpuid_count="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__get_cpuid_count" >&5 +$as_echo "$pgac_cv__get_cpuid_count" >&6; } +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + +$as_echo "#define HAVE__GET_CPUID_COUNT 1" >>confdefs.h + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuid" >&5 $as_echo_n "checking for __cpuid... " >&6; } if ${pgac_cv__cpuid+:} false; then : @@ -17438,6 +17474,175 @@ $as_echo "#define HAVE__CPUID 1" >>confdefs.h fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __cpuidex" >&5 +$as_echo_n "checking for __cpuidex... " >&6; } +if ${pgac_cv__cpuidex+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <intrin.h> +int +main () +{ +unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__cpuidex="yes" +else + pgac_cv__cpuidex="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__cpuidex" >&5 +$as_echo "$pgac_cv__cpuidex" >&6; } +if test x"$pgac_cv__cpuidex" = x"yes"; then + +$as_echo "#define HAVE__CPUIDEX 1" >>confdefs.h + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for __immintrin" >&5 +$as_echo_n "checking for __immintrin... " >&6; } +if ${pgac_cv__immintrin+:} false; then : + $as_echo_n "(cached) " >&6 +else + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +int +main () +{ +/* Don't exclude code so added return. */ + return 1701; + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv__immintrin="yes" +else + pgac_cv__immintrin="no" +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv__immintrin" >&5 +$as_echo "$pgac_cv__immintrin" >&6; } +if test x"$pgac_cv__immintrin" = x"yes"; then + +$as_echo "#define HAVE__IMMINTRIN 1" >>confdefs.h + +fi + +# Check for AVX512 intrinsics to do POPCNT calculations. +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics_+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS " +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics_=yes +else + pgac_cv_avx512_popcnt_intrinsics_=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics_" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics_" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics_" = x"yes"; then + CFLAGS_AVX512_POPCNT="" + pgac_avx512_popcnt_intrinsics=yes +fi + +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq" >&5 +$as_echo_n "checking for _mm512_popcnt_epi64 with CFLAGS=-mavx512vpopcntdq... " >&6; } +if ${pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq+:} false; then : + $as_echo_n "(cached) " >&6 +else + pgac_save_CFLAGS=$CFLAGS +CFLAGS="$pgac_save_CFLAGS -mavx512vpopcntdq" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ +#include <immintrin.h> +#include <stdint.h> +#include <stdlib.h> +#include <string.h> +int +main () +{ +const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=yes +else + pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +CFLAGS="$pgac_save_CFLAGS" +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&5 +$as_echo "$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" >&6; } +if test x"$pgac_cv_avx512_popcnt_intrinsics__mavx512vpopcntdq" = x"yes"; then + CFLAGS_AVX512_POPCNT="-mavx512vpopcntdq" + pgac_avx512_popcnt_intrinsics=yes +fi + + +$as_echo "#define HAVE__AVX512_POPCNT 1" >>confdefs.h + +fi + + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/configure.ac b/configure.ac index 57f734879e..06e8eaa6d0 100644 --- a/configure.ac +++ b/configure.ac @@ -2052,6 +2052,18 @@ if test x"$pgac_cv__get_cpuid" = x"yes"; then AC_DEFINE(HAVE__GET_CPUID, 1, [Define to 1 if you have __get_cpuid.]) fi +# Check for x86 cpuid_count instruction +AC_CACHE_CHECK([for __get_cpuid_count], [pgac_cv__get_cpuid_count], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <cpuid.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + ]])], + [pgac_cv__get_cpuid_count="yes"], + [pgac_cv__get_cpuid_count="no"])]) +if test x"$pgac_cv__get_cpuid_count" = x"yes"; then + AC_DEFINE(HAVE__GET_CPUID_COUNT, 1, [Define to 1 if you have __get_cpuid.]) +fi + AC_CACHE_CHECK([for __cpuid], [pgac_cv__cpuid], [AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], [[unsigned int exx[4] = {0, 0, 0, 0}; @@ -2063,6 +2075,38 @@ if test x"$pgac_cv__cpuid" = x"yes"; then AC_DEFINE(HAVE__CPUID, 1, [Define to 1 if you have __cpuid.]) fi +AC_CACHE_CHECK([for __cpuidex], [pgac_cv__cpuidex], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <intrin.h>], + [[unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuidex(exx[0], 7, 0); + ]])], + [pgac_cv__cpuidex="yes"], + [pgac_cv__cpuidex="no"])]) +if test x"$pgac_cv__cpuidex" = x"yes"; then + AC_DEFINE(HAVE__CPUIDEX, 1, [Define to 1 if you have __cpuidex.]) +fi + +AC_CACHE_CHECK([for __immintrin], [pgac_cv__immintrin], +[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <immintrin.h>], + [[/* Don't exclude code so added return. */ + return 1701; + ]])], + [pgac_cv__immintrin="yes"], + [pgac_cv__immintrin="no"])]) +if test x"$pgac_cv__immintrin" = x"yes"; then + AC_DEFINE(HAVE__IMMINTRIN, 1, [Define to 1 if you have immintrin.]) +fi + +# Check for AVX512 intrinsics to do POPCNT calculations. +# +PGAC_AVX512_POPCNT_INTRINSICS([]) +if test x"$pgac_avx512_popcnt_intrinsics" != x"yes"; then + PGAC_AVX512_POPCNT_INTRINSICS([-mavx512vpopcntdq]) + AC_DEFINE(HAVE__AVX512_POPCNT, 1, [Define to 1 if you have cpu + support for AVX512 POPCNT.]) +fi +AC_SUBST(CFLAGS_AVX512_POPCNT) + # Check for Intel SSE 4.2 intrinsics to do CRC calculations. # # First check if the _mm_crc32_u8 and _mm_crc32_u64 intrinsics can be used diff --git a/meson.build b/meson.build index c8fdfeb0ec..cd508096e5 100644 --- a/meson.build +++ b/meson.build @@ -1783,6 +1783,37 @@ elif cc.links(''' endif +if cc.links(''' + #include <cpuid.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __get_cpuid_count(7, 0, &exx[0], &exx[1], &exx[2], &exx[3]); + } + ''', name: '__get_cpuid_count', + args: test_c_args) + cdata.set('HAVE__GET_CPUID_COUNT', 1) +elif cc.links(''' + #include <intrin.h> + int main(int arg, char **argv) + { + unsigned int exx[4] = {0, 0, 0, 0}; + __cpuidex(exx, 7, 0); + } + ''', name: '__cpuidex', + args: test_c_args) + cdata.set('HAVE__CPUIDEX', 1) +endif + + +# Check for header immintrin.h +if cc.has_header('immintrin.h', + include_directories: postgres_inc, args: test_c_args) + cdata.set('HAVE__IMMINTRIN', 1, + description: 'Define to 1 if you have the immintrin.h header file.') +endif + + # Defend against clang being used on x86-32 without SSE2 enabled. As current # versions of clang do not understand -fexcess-precision=standard, the use of # x87 floating point operations leads to problems like isinf possibly returning @@ -2158,6 +2189,47 @@ endif +############################################################### +# AVX 512 POPCNT Intrinsic check +############################################################### +have_avx512_popcnt = false +cflags_avx512_popcnt = [] +if host_cpu == 'x86_64' + test_flags = ['-mavx512vpopcntdq'] + if host_system == 'windows' + test_flags = ['/arch:AVX512'] + endif + prog = ''' + #include <immintrin.h> + #include <stdint.h> + #include <stdlib.h> + #include <string.h> + void main(void) + { + const uint64_t *buf = malloc((size_t)64); + uint64_t popcnt = 0; + __m512i accumulator = _mm512_setzero_si512(); + const __m512i v = _mm512_loadu_si512((const __m512i *)buf); + const __m512i p = _mm512_popcnt_epi64(v); + memset(buf, 0, 64); + accumulator = _mm512_add_epi64(accumulator, p); + popcnt = _mm512_reduce_add_epi64(accumulator); + /* return computed value, to prevent the above being optimized away */ + return popcnt == 0; + }''' + if cc.links(prog, name: '_mm512_* methods with -mavx512vpopcntdq flag.', + args: test_c_args + test_flags) + have_avx512_popcnt = true + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cdata.set('HAVE__AVX512_POPCNT', 1) + cflags_avx512_popcnt = test_flags + else + have_avx512_popcnt = false + cdata.set('USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 1) + cflags_avx512_popcnt = [] + endif # compile/link test +endif # host_cpu check + ############################################################### # Library / OS tests ############################################################### diff --git a/src/Makefile.global.in b/src/Makefile.global.in index 8b3f8c24e0..089f49b7f3 100644 --- a/src/Makefile.global.in +++ b/src/Makefile.global.in @@ -263,6 +263,7 @@ CXXFLAGS_SL_MODULE = @CXXFLAGS_SL_MODULE@ CFLAGS_UNROLL_LOOPS = @CFLAGS_UNROLL_LOOPS@ CFLAGS_VECTORIZE = @CFLAGS_VECTORIZE@ CFLAGS_CRC = @CFLAGS_CRC@ +CFLAGS_AVX512_POPCNT = @CFLAGS_AVX512_POPCNT@ PERMIT_DECLARATION_AFTER_STATEMENT = @PERMIT_DECLARATION_AFTER_STATEMENT@ CXXFLAGS = @CXXFLAGS@ diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 591e1ca3df..33a831e768 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -558,6 +558,18 @@ /* Define to 1 if you have __get_cpuid. */ #undef HAVE__GET_CPUID +/* Define to 1 if you have __get_cpuid_count. */ +#undef HAVE__GET_CPUID_COUNT + +/* Define to 1 if you have __get_cpuidex. */ +#undef HAVE__GET_CPUIDEX + +/* Define to 1 if you have immintrin. */ +#undef HAVE__IMMINTRIN + +/* Define to 1 if you have AVX512. */ +#undef HAVE__AVX512_POPCNT + /* Define to 1 if your compiler understands _Static_assert. */ #undef HAVE__STATIC_ASSERT diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 26f6a48377..8215316b0e 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -306,8 +306,8 @@ extern PGDLLIMPORT uint64 (*pg_popcount) (const char *buf, int bytes); #else /* - * Use a portable implementation -- no need for a function pointer. Use - * inlining for small speed increase. + * Use a portable implementation -- no need for a function pointer. Use + * inlining for small speed increase. */ extern int pg_popcount32(uint32 word); extern int pg_popcount64(uint64 word); diff --git a/src/makefiles/meson.build b/src/makefiles/meson.build index b0f4178b3d..ee3647282e 100644 --- a/src/makefiles/meson.build +++ b/src/makefiles/meson.build @@ -100,6 +100,7 @@ pgxs_kv = { ' '.join(cflags_no_decl_after_statement), 'CFLAGS_CRC': ' '.join(cflags_crc), + 'CFLAGS_AVX512_POPCNT': ' '.join(cflags_avx512_popcnt), 'CFLAGS_UNROLL_LOOPS': ' '.join(unroll_loops_cflags), 'CFLAGS_VECTORIZE': ' '.join(vectorize_cflags), diff --git a/src/port/Makefile b/src/port/Makefile index 1499985dfc..92bd419953 100644 --- a/src/port/Makefile +++ b/src/port/Makefile @@ -89,6 +89,11 @@ pg_crc32c_sse42.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_shlib.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_sse42_srv.o: CFLAGS+=$(CFLAGS_CRC) +# Newer processors can use AVX-512 POPCNT Capabilities +pg_popcount_x86_64_accel.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_shlib.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) +pg_popcount_x86_64_accel_srv.o: CFLAGS+=$(CFLAGS_AVX512_POPCNT) + # all versions of pg_crc32c_armv8.o need CFLAGS_CRC pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC) pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC) diff --git a/src/port/meson.build b/src/port/meson.build index cf6e9fa06c..0647e7a4f7 100644 --- a/src/port/meson.build +++ b/src/port/meson.build @@ -8,7 +8,6 @@ pgport_sources = [ 'path.c', 'pg_bitutils.c', 'pg_popcount_x86_64_choose.c', - 'pg_popcount_x86_64_accel.c', 'pg_strong_random.c', 'pgcheckdir.c', 'pgmkdirp.c', @@ -86,6 +85,7 @@ replace_funcs_pos = [ ['pg_crc32c_sse42', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK', 'crc'], ['pg_crc32c_sse42_choose', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], ['pg_crc32c_sb8', 'USE_SSE42_CRC32C_WITH_RUNTIME_CHECK'], + ['pg_popcount_x86_64_accel', 'USE_AVX512_POPCNT_WITH_RUNTIME_CHECK', 'avx512'], # arm / aarch64 ['pg_crc32c_armv8', 'USE_ARMV8_CRC32C'], @@ -100,8 +100,8 @@ replace_funcs_pos = [ ['pg_crc32c_sb8', 'USE_SLICING_BY_8_CRC32C'], ] -pgport_cflags = {'crc': cflags_crc} -pgport_sources_cflags = {'crc': []} +pgport_cflags = {'crc': cflags_crc, 'avx512': cflags_avx512_popcnt} +pgport_sources_cflags = {'crc': [], 'avx512': []} foreach f : replace_funcs_neg func = f.get(0) -- 2.34.1 ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Popcount optimization using AVX512 @ 2024-03-25 15:12 Tom Lane <[email protected]> parent: Amonson, Paul D <[email protected]> 0 siblings, 2 replies; 12+ messages in thread From: Tom Lane @ 2024-03-25 15:12 UTC (permalink / raw) To: Amonson, Paul D <[email protected]>; +Cc: David Rowley <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> "Amonson, Paul D" <[email protected]> writes: > I am re-posting the patches as CI for Mac failed (CI error not code/test error). The patches are the same as last time. Just for a note --- the cfbot will re-test existing patches every so often without needing a bump. The current cycle period seems to be about two days. regards, tom lane ^ permalink raw reply [nested|flat] 12+ messages in thread
* RE: Popcount optimization using AVX512 @ 2024-03-25 15:20 Amonson, Paul D <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 1 reply; 12+ messages in thread From: Amonson, Paul D @ 2024-03-25 15:20 UTC (permalink / raw) To: Tom Lane <[email protected]>; +Cc: David Rowley <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> > -----Original Message----- > From: Tom Lane <[email protected]> > Sent: Monday, March 25, 2024 8:12 AM > To: Amonson, Paul D <[email protected]> > Cc: David Rowley <[email protected]>; Nathan Bossart > Subject: Re: Popcount optimization using AVX512 >... > Just for a note --- the cfbot will re-test existing patches every so often without > needing a bump. The current cycle period seems to be about two days. > > regards, tom lane Good to know! Maybe this is why I thought it originally passed CI and suddenly this morning there is a failure. I noticed at least 2 other patch runs also failed in the same way. Thanks, Paul ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Popcount optimization using AVX512 @ 2024-03-25 15:44 Joe Conway <[email protected]> parent: Tom Lane <[email protected]> 1 sibling, 0 replies; 12+ messages in thread From: Joe Conway @ 2024-03-25 15:44 UTC (permalink / raw) To: Tom Lane <[email protected]>; Amonson, Paul D <[email protected]>; +Cc: David Rowley <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]>; Thomas Munro <[email protected]> On 3/25/24 11:12, Tom Lane wrote: > "Amonson, Paul D" <[email protected]> writes: >> I am re-posting the patches as CI for Mac failed (CI error not code/test error). The patches are the same as last time. > > Just for a note --- the cfbot will re-test existing patches every > so often without needing a bump. The current cycle period seems to > be about two days. Just an FYI -- there seems to be an issue with all three of the macos cfbot runners (mine included). I spent time over the weekend working with Thomas Munro (added to CC list) trying different fixes to no avail. Help from macos CI wizards would be gratefully accepted... -- Joe Conway PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 12+ messages in thread
* RE: Popcount optimization using AVX512 @ 2024-03-25 18:42 Amonson, Paul D <[email protected]> parent: Amonson, Paul D <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: Amonson, Paul D @ 2024-03-25 18:42 UTC (permalink / raw) To: Amonson, Paul D <[email protected]>; Tom Lane <[email protected]>; +Cc: David Rowley <[email protected]>; Nathan Bossart <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> > -----Original Message----- > From: Amonson, Paul D <[email protected]> > Sent: Monday, March 25, 2024 8:20 AM > To: Tom Lane <[email protected]> > Cc: David Rowley <[email protected]>; Nathan Bossart > <[email protected]>; Andres Freund <[email protected]>; Alvaro > Herrera <[email protected]>; Shankaran, Akash > <[email protected]>; Noah Misch <[email protected]>; Matthias > van de Meent <[email protected]>; pgsql- > [email protected] > Subject: RE: Popcount optimization using AVX512 > Ok, CI turned green after my re-post of the patches. Can this please get merged? Thanks, Paul ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Popcount optimization using AVX512 @ 2024-03-25 20:05 Nathan Bossart <[email protected]> parent: Amonson, Paul D <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Nathan Bossart @ 2024-03-25 20:05 UTC (permalink / raw) To: Amonson, Paul D <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; Andres Freund <[email protected]>; Alvaro Herrera <[email protected]>; Shankaran, Akash <[email protected]>; Noah Misch <[email protected]>; Matthias van de Meent <[email protected]>; [email protected] <[email protected]> On Mon, Mar 25, 2024 at 06:42:36PM +0000, Amonson, Paul D wrote: > Ok, CI turned green after my re-post of the patches. Can this please get > merged? Thanks for the new patches. I intend to take another look soon. -- Nathan Bossart Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2024-03-25 20:05 UTC | newest] Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-01-17 06:53 [PATCH 1/8] cirrus: include hints how to install OS packages.. Justin Pryzby <[email protected]> 2024-03-19 22:56 RE: Popcount optimization using AVX512 Amonson, Paul D <[email protected]> 2024-03-20 04:26 ` Re: Popcount optimization using AVX512 David Rowley <[email protected]> 2024-03-20 14:23 ` RE: Popcount optimization using AVX512 Amonson, Paul D <[email protected]> 2024-03-21 00:28 ` Re: Popcount optimization using AVX512 David Rowley <[email protected]> 2024-03-21 19:17 ` RE: Popcount optimization using AVX512 Amonson, Paul D <[email protected]> 2024-03-25 15:06 ` RE: Popcount optimization using AVX512 Amonson, Paul D <[email protected]> 2024-03-25 15:12 ` Re: Popcount optimization using AVX512 Tom Lane <[email protected]> 2024-03-25 15:20 ` RE: Popcount optimization using AVX512 Amonson, Paul D <[email protected]> 2024-03-25 18:42 ` RE: Popcount optimization using AVX512 Amonson, Paul D <[email protected]> 2024-03-25 20:05 ` Re: Popcount optimization using AVX512 Nathan Bossart <[email protected]> 2024-03-25 15:44 ` Re: Popcount optimization using AVX512 Joe Conway <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox