agora inbox for pgsql-hackers@postgresql.orghelp / color / mirror / Atom feed
[PATCH v8 1/3] Remove some unnecessary optimizations in popcount code. 8+ messages / 1 participants [nested] [flat]
* [PATCH v8 1/3] Remove some unnecessary optimizations in popcount code. @ 2026-01-22 17:33 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-01-22 17:33 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. This commit does the following: * Removes the code in pg_popcount() and pg_popcount_masked() that sets the function pointer threshold based on SIDEOF_VOID_P. Consequently, 32-bit builds should follow the inline path for inputs between 4-8 bytes instead of calling pg_popcount_optimized() (which is probably just calling pg_popcount_portable(), anyway). While it is possible that this results in a small regression for those inputs on 32-bit builds, it seems unlikely to produce noticeable performance differences on those machines. Furthermore, I found no evidence of benchmarks for this area of code for 32-bit builds. * Removes the 32-bit optimizations in pg_popcount_portable() and pg_popcount_masked_portable(). This means that 32-bit builds instead use a simple while loop. As above, we are not too concerned about regressions on 32-bit machines. * Removes 32-bit optimizations in pg_popcount_x86.c. This is dead code because everything in this file is only compiled when HAVE_X86_64_POPCNTQ is defined, and that macro is only defined for x86-64. * Removes alignment checks in pg_popcount_sse42() and pg_popcount_masked_sse42(). These are unnecessary for x86, and it's unclear whether they make any meaningful performance difference. Since we allow misaligned accesses now, this commit also adds pg_attribute_no_sanitize_alignment() to these functions. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 24 +----------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 107 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..c3049d71894 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -329,17 +329,7 @@ extern uint64 pg_popcount_masked_optimized(const char *buf, int bytes, bits8 mas static inline uint64 pg_popcount(const char *buf, int bytes) { - /* - * We set the threshold to the point at which we'll first use special - * instructions in the optimized version. - */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -360,17 +350,7 @@ pg_popcount(const char *buf, int bytes) static inline uint64 pg_popcount_masked(const char *buf, int bytes, bits8 mask) { - /* - * We set the threshold to the point at which we'll first use special - * instructions in the optimized version. - */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --dIDo5IfS/hVe25hD Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v8-0002-Remove-specialized-word-length-popcount-implement.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v9 1/3] Remove some unnecessary optimizations in popcount code. @ 2026-01-22 17:33 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-01-22 17:33 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. This commit does the following: * Removes the code in pg_popcount() and pg_popcount_masked() that sets the function pointer threshold based on SIDEOF_VOID_P. Consequently, 32-bit builds should follow the inline path for inputs between 4-8 bytes instead of calling pg_popcount_optimized() (which is probably just calling pg_popcount_portable(), anyway). While it is possible that this results in a small regression for those inputs on 32-bit builds, it seems unlikely to produce noticeable performance differences on those machines. Furthermore, I found no evidence of benchmarks for this area of code for 32-bit builds. * Removes the 32-bit optimizations in pg_popcount_portable() and pg_popcount_masked_portable(). This means that 32-bit builds instead use a simple while loop. As above, we are not too concerned about regressions on 32-bit machines. * Removes 32-bit optimizations in pg_popcount_x86.c. This is dead code because everything in this file is only compiled when HAVE_X86_64_POPCNTQ is defined, and that macro is only defined for x86-64. * Removes alignment checks in pg_popcount_sse42() and pg_popcount_masked_sse42(). These are unnecessary for x86, and it's unclear whether they make any meaningful performance difference. Since we allow misaligned accesses now, this commit also adds pg_attribute_no_sanitize_alignment() to these functions. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 24 +----------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 107 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..c3049d71894 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -329,17 +329,7 @@ extern uint64 pg_popcount_masked_optimized(const char *buf, int bytes, bits8 mas static inline uint64 pg_popcount(const char *buf, int bytes) { - /* - * We set the threshold to the point at which we'll first use special - * instructions in the optimized version. - */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -360,17 +350,7 @@ pg_popcount(const char *buf, int bytes) static inline uint64 pg_popcount_masked(const char *buf, int bytes, bits8 mask) { - /* - * We set the threshold to the point at which we'll first use special - * instructions in the optimized version. - */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --oEt0RkbzovU82iL7 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v9-0002-Remove-specialized-word-length-popcount-implement.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v10 1/3] Remove some unnecessary optimizations in popcount code. @ 2026-01-22 17:33 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-01-22 17:33 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. This commit does the following: * Removes the code in pg_popcount() and pg_popcount_masked() that sets the function pointer threshold based on SIDEOF_VOID_P. Consequently, 32-bit builds should follow the inline path for inputs between 4-8 bytes instead of calling pg_popcount_optimized() (which is probably just calling pg_popcount_portable(), anyway). While it is possible that this results in a small regression for those inputs on 32-bit builds, it seems unlikely to produce noticeable performance differences on those machines. Furthermore, I found no evidence of benchmarks for this area of code for 32-bit builds. * Removes the 32-bit optimizations in pg_popcount_portable() and pg_popcount_masked_portable(). This means that 32-bit builds instead use a simple while loop. As above, we are not too concerned about regressions on 32-bit machines. * Removes 32-bit optimizations in pg_popcount_x86.c. This is dead code because everything in this file is only compiled when HAVE_X86_64_POPCNTQ is defined, and that macro is only defined for x86-64. * Removes alignment checks in pg_popcount_sse42() and pg_popcount_masked_sse42(). These are unnecessary for x86, and it's unclear whether they make any meaningful performance difference. Since we allow misaligned accesses now, this commit also adds pg_attribute_no_sanitize_alignment() to these functions. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 16 +------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 99 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..20c11b79c61 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -333,13 +333,7 @@ pg_popcount(const char *buf, int bytes) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -364,13 +358,7 @@ pg_popcount_masked(const char *buf, int bytes, bits8 mask) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --rlwGoOKBy5dABfM4 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v10-0002-Remove-specialized-word-length-popcount-implemen.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v13 1/5] Remove some unnecessary optimizations in popcount code. @ 2026-02-06 15:26 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-02-06 15:26 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 16 +------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 99 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..20c11b79c61 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -333,13 +333,7 @@ pg_popcount(const char *buf, int bytes) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -364,13 +358,7 @@ pg_popcount_masked(const char *buf, int bytes, bits8 mask) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --5NNwvjHYUnRaoEDo Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v13-0002-Remove-specialized-word-length-popcount-implemen.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v11 1/4] Remove some unnecessary optimizations in popcount code. @ 2026-02-06 15:26 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-02-06 15:26 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. This commit does the following: * Removes the code in pg_popcount() and pg_popcount_masked() that sets the function pointer threshold based on SIDEOF_VOID_P. Consequently, 32-bit builds should follow the inline path for inputs between 4-8 bytes instead of calling pg_popcount_optimized() (which is probably just calling pg_popcount_portable(), anyway). While it is possible that this results in a small regression for those inputs on 32-bit builds, it seems unlikely to produce noticeable performance differences on those machines. Furthermore, I found no evidence of past benchmarks for this area of code for 32-bit builds. * Removes the 32-bit optimizations in pg_popcount_portable() and pg_popcount_masked_portable(). This means that 32-bit builds instead use a simple while loop. As above, we are not too concerned about regressions on 32-bit machines. * Removes 32-bit optimizations in pg_popcount_x86.c. This is dead code because everything in this file is only compiled when HAVE_X86_64_POPCNTQ is defined, and that macro is only defined for x86-64. * Removes alignment checks in pg_popcount_sse42() and pg_popcount_masked_sse42(). These are unnecessary for x86, and it's unclear whether they make any meaningful performance difference. Since we allow misaligned accesses now, this commit also adds pg_attribute_no_sanitize_alignment() to these functions. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 16 +------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 99 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..20c11b79c61 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -333,13 +333,7 @@ pg_popcount(const char *buf, int bytes) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -364,13 +358,7 @@ pg_popcount_masked(const char *buf, int bytes, bits8 mask) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --mih0IfQp6StcW7xc Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v11-0002-Remove-specialized-word-length-popcount-implemen.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v11 1/4] Remove some unnecessary optimizations in popcount code. @ 2026-02-06 15:26 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-02-06 15:26 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. This commit does the following: * Removes the code in pg_popcount() and pg_popcount_masked() that sets the function pointer threshold based on SIDEOF_VOID_P. Consequently, 32-bit builds should follow the inline path for inputs between 4-8 bytes instead of calling pg_popcount_optimized() (which is probably just calling pg_popcount_portable(), anyway). While it is possible that this results in a small regression for those inputs on 32-bit builds, it seems unlikely to produce noticeable performance differences on those machines. Furthermore, I found no evidence of past benchmarks for this area of code for 32-bit builds. * Removes the 32-bit optimizations in pg_popcount_portable() and pg_popcount_masked_portable(). This means that 32-bit builds instead use a simple while loop. As above, we are not too concerned about regressions on 32-bit machines. * Removes 32-bit optimizations in pg_popcount_x86.c. This is dead code because everything in this file is only compiled when HAVE_X86_64_POPCNTQ is defined, and that macro is only defined for x86-64. * Removes alignment checks in pg_popcount_sse42() and pg_popcount_masked_sse42(). These are unnecessary for x86, and it's unclear whether they make any meaningful performance difference. Since we allow misaligned accesses now, this commit also adds pg_attribute_no_sanitize_alignment() to these functions. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 16 +------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 99 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..20c11b79c61 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -333,13 +333,7 @@ pg_popcount(const char *buf, int bytes) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -364,13 +358,7 @@ pg_popcount_masked(const char *buf, int bytes, bits8 mask) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --mih0IfQp6StcW7xc Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v11-0002-Remove-specialized-word-length-popcount-implemen.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v12 1/4] Remove some unnecessary optimizations in popcount code. @ 2026-02-06 15:26 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-02-06 15:26 UTC (permalink / raw) Over the past few releases, we've added a huge amount of complexity to our popcount implementations. Commits fbe327e5b4, 79e232ca01, 8c6653516c, and 25dc485074 did some preliminary refactoring, but many opportunities remain. In particular, if we disclaim interest in micro-optimizing this code for 32-bit builds and in unproven alignment checks, we can remove a decent chunk of code. Suggested-by: John Naylor <johncnaylorls@gmail.com> Reviewed-by: John Naylor <johncnaylorls@gmail.com> Discussion: https://postgr.es/m/CANWCAZY7R%2Biy%2Br9YM_sySNydHzNqUirx1xk0tB3ej5HO62GdgQ%40mail.gmail.com --- src/include/port/pg_bitutils.h | 16 +------- src/port/pg_bitutils.c | 30 --------------- src/port/pg_popcount_x86.c | 67 ++++++---------------------------- 3 files changed, 14 insertions(+), 99 deletions(-) diff --git a/src/include/port/pg_bitutils.h b/src/include/port/pg_bitutils.h index 35761f509ec..20c11b79c61 100644 --- a/src/include/port/pg_bitutils.h +++ b/src/include/port/pg_bitutils.h @@ -333,13 +333,7 @@ pg_popcount(const char *buf, int bytes) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; @@ -364,13 +358,7 @@ pg_popcount_masked(const char *buf, int bytes, bits8 mask) * We set the threshold to the point at which we'll first use special * instructions in the optimized version. */ -#if SIZEOF_VOID_P >= 8 - int threshold = 8; -#else - int threshold = 4; -#endif - - if (bytes < threshold) + if (bytes < 8) { uint64 popcnt = 0; diff --git a/src/port/pg_bitutils.c b/src/port/pg_bitutils.c index ffda75825e5..bec06c06fc3 100644 --- a/src/port/pg_bitutils.c +++ b/src/port/pg_bitutils.c @@ -167,20 +167,6 @@ pg_popcount_portable(const char *buf, int bytes) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++); - bytes -= 4; - } - buf = (const char *) words; } #endif @@ -215,22 +201,6 @@ pg_popcount_masked_portable(const char *buf, int bytes, bits8 mask) bytes -= 8; } - buf = (const char *) words; - } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_portable(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; } #endif diff --git a/src/port/pg_popcount_x86.c b/src/port/pg_popcount_x86.c index 245f0167d00..7aebf69898b 100644 --- a/src/port/pg_popcount_x86.c +++ b/src/port/pg_popcount_x86.c @@ -376,40 +376,20 @@ __asm__ __volatile__(" popcntq %1,%0\n":"=q"(res):"rm"(word):"cc"); * pg_popcount_sse42 * Returns the number of 1-bits in buf */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_sse42(const char *buf, int bytes) { uint64 popcnt = 0; + const uint64 *words = (const uint64 *) buf; -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++); - bytes -= 4; - } - - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) @@ -422,44 +402,21 @@ pg_popcount_sse42(const char *buf, int bytes) * pg_popcount_masked_sse42 * Returns the number of 1-bits in buf after applying the mask to each byte */ +pg_attribute_no_sanitize_alignment() static uint64 pg_popcount_masked_sse42(const char *buf, int bytes, bits8 mask) { uint64 popcnt = 0; - -#if SIZEOF_VOID_P >= 8 - /* Process in 64-bit chunks if the buffer is aligned */ uint64 maskv = ~UINT64CONST(0) / 0xFF * mask; + const uint64 *words = (const uint64 *) buf; - if (buf == (const char *) TYPEALIGN(8, buf)) + while (bytes >= 8) { - const uint64 *words = (const uint64 *) buf; - - while (bytes >= 8) - { - popcnt += pg_popcount64_sse42(*words++ & maskv); - bytes -= 8; - } - - buf = (const char *) words; + popcnt += pg_popcount64_sse42(*words++ & maskv); + bytes -= 8; } -#else - /* Process in 32-bit chunks if the buffer is aligned. */ - uint32 maskv = ~((uint32) 0) / 0xFF * mask; - - if (buf == (const char *) TYPEALIGN(4, buf)) - { - const uint32 *words = (const uint32 *) buf; - - while (bytes >= 4) - { - popcnt += pg_popcount32_sse42(*words++ & maskv); - bytes -= 4; - } - buf = (const char *) words; - } -#endif + buf = (const char *) words; /* Process any remaining bytes */ while (bytes--) -- 2.50.1 (Apple Git-155) --AFLuD1w+kMRlv6Zk Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=v12-0002-Remove-specialized-word-length-popcount-implemen.patch ^ permalink raw reply [nested|flat] 8+ messages in thread
* [PATCH v1 1/1] Remove unnecessary volatile qualifiers. @ 2026-06-30 21:41 Nathan Bossart <nathan@postgresql.org> 0 siblings, 0 replies; 8+ messages in thread From: Nathan Bossart @ 2026-06-30 21:41 UTC (permalink / raw) This commit cleans up volatile qualifiers that fit the below criteria: * Accesses to shared memory protected by a spinlock or LWLock. Before commit 0709b7ee72, callers had to use volatile when accessing spinlock-protected shared memory. Since spinlock acquire/release became compiler barriers, and because LWLocks provide the same guarantee, that is no longer necessary. These either predate that change or were cargo-culted from code that did. * Pointers to pg_atomic_* variables or local variables that hold a value returned by a pg_atomic_* function. The pointer arguments for the pg_atomic_* functions are volatile-qualified, so there's no need to mark the pointer volatile. Likewise, a local variable that just holds the result of a pg_atomic_* function gains nothing from volatile. * Accesses to struct members that are marked volatile in the struct definition. There's no need to mark these pointers volatile, either. * Leftovers from removed PG_TRY blocks. These were marked volatile when they were modified inside a PG_TRY block and used afterward, but the PG_TRY was later removed. --- src/backend/access/transam/clog.c | 2 +- src/backend/catalog/index.c | 2 +- src/backend/commands/async.c | 4 ++-- src/backend/replication/syncrep.c | 19 +++++++--------- src/backend/storage/ipc/procsignal.c | 10 ++++----- src/backend/storage/ipc/shm_toc.c | 31 ++++++++++++--------------- src/backend/storage/lmgr/lock.c | 2 +- src/backend/storage/lmgr/proc.c | 3 +-- src/test/modules/test_shm_mq/setup.c | 4 ++-- src/test/modules/test_shm_mq/worker.c | 2 +- 10 files changed, 36 insertions(+), 43 deletions(-) diff --git a/src/backend/access/transam/clog.c b/src/backend/access/transam/clog.c index 75012d4b8f0..47975ba892f 100644 --- a/src/backend/access/transam/clog.c +++ b/src/backend/access/transam/clog.c @@ -450,7 +450,7 @@ static bool TransactionGroupUpdateXidStatus(TransactionId xid, XidStatus status, XLogRecPtr lsn, int64 pageno) { - volatile PROC_HDR *procglobal = ProcGlobal; + PROC_HDR *procglobal = ProcGlobal; PGPROC *proc = MyProc; uint32 nextidx; uint32 wakeidx; diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 9407c357f27..81bba4beac7 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -3637,7 +3637,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, int save_sec_context; int save_nestlevel; IndexInfo *indexInfo; - volatile bool skipped_constraint = false; + bool skipped_constraint = false; PGRUsage ru0; bool progress = ((params->options & REINDEXOPT_REPORT_PROGRESS) != 0); bool set_tablespace = false; diff --git a/src/backend/commands/async.c b/src/backend/commands/async.c index eee8bc29f38..2799f989b96 100644 --- a/src/backend/commands/async.c +++ b/src/backend/commands/async.c @@ -605,7 +605,7 @@ static void CleanupListenersOnExit(void); static bool IsListeningOn(const char *channel); static void asyncQueueUnregister(void); static bool asyncQueueIsFull(void); -static bool asyncQueueAdvance(volatile QueuePosition *position, int entryLength); +static bool asyncQueueAdvance(QueuePosition *position, int entryLength); static void asyncQueueNotificationToEntry(Notification *n, AsyncQueueEntry *qe); static ListCell *asyncQueueAddEntries(ListCell *nextNotify); static double asyncQueueUsage(void); @@ -1968,7 +1968,7 @@ asyncQueueIsFull(void) * returns true, else false. */ static bool -asyncQueueAdvance(volatile QueuePosition *position, int entryLength) +asyncQueueAdvance(QueuePosition *position, int entryLength) { int64 pageno = QUEUE_POS_PAGE(*position); int offset = QUEUE_POS_OFFSET(*position); diff --git a/src/backend/replication/syncrep.c b/src/backend/replication/syncrep.c index e0e30579c59..d870f09e0a0 100644 --- a/src/backend/replication/syncrep.c +++ b/src/backend/replication/syncrep.c @@ -483,7 +483,6 @@ SyncRepInitConfig(void) void SyncRepReleaseWaiters(void) { - volatile WalSndCtlData *walsndctl = WalSndCtl; XLogRecPtr writePtr; XLogRecPtr flushPtr; XLogRecPtr applyPtr; @@ -558,19 +557,19 @@ SyncRepReleaseWaiters(void) * Set the lsn first so that when we wake backends they will release up to * this location. */ - if (walsndctl->lsn[SYNC_REP_WAIT_WRITE] < writePtr) + if (WalSndCtl->lsn[SYNC_REP_WAIT_WRITE] < writePtr) { - walsndctl->lsn[SYNC_REP_WAIT_WRITE] = writePtr; + WalSndCtl->lsn[SYNC_REP_WAIT_WRITE] = writePtr; numwrite = SyncRepWakeQueue(false, SYNC_REP_WAIT_WRITE); } - if (walsndctl->lsn[SYNC_REP_WAIT_FLUSH] < flushPtr) + if (WalSndCtl->lsn[SYNC_REP_WAIT_FLUSH] < flushPtr) { - walsndctl->lsn[SYNC_REP_WAIT_FLUSH] = flushPtr; + WalSndCtl->lsn[SYNC_REP_WAIT_FLUSH] = flushPtr; numflush = SyncRepWakeQueue(false, SYNC_REP_WAIT_FLUSH); } - if (walsndctl->lsn[SYNC_REP_WAIT_APPLY] < applyPtr) + if (WalSndCtl->lsn[SYNC_REP_WAIT_APPLY] < applyPtr) { - walsndctl->lsn[SYNC_REP_WAIT_APPLY] = applyPtr; + WalSndCtl->lsn[SYNC_REP_WAIT_APPLY] = applyPtr; numapply = SyncRepWakeQueue(false, SYNC_REP_WAIT_APPLY); } @@ -777,8 +776,7 @@ SyncRepGetCandidateStandbys(SyncRepStandbyData **standbys) n = 0; for (i = 0; i < max_wal_senders; i++) { - volatile WalSnd *walsnd; /* Use volatile pointer to prevent code - * rearrangement */ + WalSnd *walsnd; SyncRepStandbyData *stby; WalSndState state; /* not included in SyncRepStandbyData */ @@ -915,7 +913,6 @@ SyncRepGetStandbyPriority(void) static int SyncRepWakeQueue(bool all, int mode) { - volatile WalSndCtlData *walsndctl = WalSndCtl; int numprocs = 0; dlist_mutable_iter iter; @@ -930,7 +927,7 @@ SyncRepWakeQueue(bool all, int mode) /* * Assume the queue is ordered by LSN */ - if (!all && walsndctl->lsn[mode] < proc->waitLSN) + if (!all && WalSndCtl->lsn[mode] < proc->waitLSN) return numprocs; /* diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c index 1397f65f67b..4acef19e563 100644 --- a/src/backend/storage/ipc/procsignal.c +++ b/src/backend/storage/ipc/procsignal.c @@ -295,7 +295,7 @@ CleanupProcSignalState(int status, Datum arg) int SendProcSignal(pid_t pid, ProcSignalReason reason, ProcNumber procNumber) { - volatile ProcSignalSlot *slot; + ProcSignalSlot *slot; if (procNumber != INVALID_PROC_NUMBER) { @@ -380,7 +380,7 @@ EmitProcSignalBarrier(ProcSignalBarrierType type) */ for (int i = 0; i < NumProcSignalSlots; i++) { - volatile ProcSignalSlot *slot = &ProcSignal->psh_slot[i]; + ProcSignalSlot *slot = &ProcSignal->psh_slot[i]; pg_atomic_fetch_or_u32(&slot->pss_barrierCheckMask, flagbit); } @@ -406,7 +406,7 @@ EmitProcSignalBarrier(ProcSignalBarrierType type) */ for (int i = NumProcSignalSlots - 1; i >= 0; i--) { - volatile ProcSignalSlot *slot = &ProcSignal->psh_slot[i]; + ProcSignalSlot *slot = &ProcSignal->psh_slot[i]; pid_t pid = pg_atomic_read_u32(&slot->pss_pid); if (pid != 0) @@ -512,7 +512,7 @@ ProcessProcSignalBarrier(void) { uint64 local_gen; uint64 shared_gen; - volatile uint32 flags; + uint32 flags; Assert(MyProcSignalSlot); @@ -670,7 +670,7 @@ ResetProcSignalBarrierBits(uint32 flags) static bool CheckProcSignal(ProcSignalReason reason) { - volatile ProcSignalSlot *slot = MyProcSignalSlot; + ProcSignalSlot *slot = MyProcSignalSlot; if (slot != NULL) { diff --git a/src/backend/storage/ipc/shm_toc.c b/src/backend/storage/ipc/shm_toc.c index 2f9fbb0a519..2217d48c3d9 100644 --- a/src/backend/storage/ipc/shm_toc.c +++ b/src/backend/storage/ipc/shm_toc.c @@ -87,7 +87,6 @@ shm_toc_attach(uint64 magic, void *address) void * shm_toc_allocate(shm_toc *toc, Size nbytes) { - volatile shm_toc *vtoc = toc; Size total_bytes; Size allocated_bytes; Size nentry; @@ -103,9 +102,9 @@ shm_toc_allocate(shm_toc *toc, Size nbytes) SpinLockAcquire(&toc->toc_mutex); - total_bytes = vtoc->toc_total_bytes; - allocated_bytes = vtoc->toc_allocated_bytes; - nentry = vtoc->toc_nentry; + total_bytes = toc->toc_total_bytes; + allocated_bytes = toc->toc_allocated_bytes; + nentry = toc->toc_nentry; toc_bytes = offsetof(shm_toc, toc_entry) + nentry * sizeof(shm_toc_entry) + allocated_bytes; @@ -117,7 +116,7 @@ shm_toc_allocate(shm_toc *toc, Size nbytes) (errcode(ERRCODE_OUT_OF_MEMORY), errmsg("out of shared memory"))); } - vtoc->toc_allocated_bytes += nbytes; + toc->toc_allocated_bytes += nbytes; SpinLockRelease(&toc->toc_mutex); @@ -130,16 +129,15 @@ shm_toc_allocate(shm_toc *toc, Size nbytes) Size shm_toc_freespace(shm_toc *toc) { - volatile shm_toc *vtoc = toc; Size total_bytes; Size allocated_bytes; Size nentry; Size toc_bytes; SpinLockAcquire(&toc->toc_mutex); - total_bytes = vtoc->toc_total_bytes; - allocated_bytes = vtoc->toc_allocated_bytes; - nentry = vtoc->toc_nentry; + total_bytes = toc->toc_total_bytes; + allocated_bytes = toc->toc_allocated_bytes; + nentry = toc->toc_nentry; SpinLockRelease(&toc->toc_mutex); toc_bytes = offsetof(shm_toc, toc_entry) + nentry * sizeof(shm_toc_entry); @@ -170,7 +168,6 @@ shm_toc_freespace(shm_toc *toc) void shm_toc_insert(shm_toc *toc, uint64 key, void *address) { - volatile shm_toc *vtoc = toc; Size total_bytes; Size allocated_bytes; Size nentry; @@ -183,14 +180,14 @@ shm_toc_insert(shm_toc *toc, uint64 key, void *address) SpinLockAcquire(&toc->toc_mutex); - total_bytes = vtoc->toc_total_bytes; - allocated_bytes = vtoc->toc_allocated_bytes; - nentry = vtoc->toc_nentry; + total_bytes = toc->toc_total_bytes; + allocated_bytes = toc->toc_allocated_bytes; + nentry = toc->toc_nentry; #ifdef USE_ASSERT_CHECKING /* Verify no duplicate keys */ for (Size i = 0; i < nentry; i++) - Assert(vtoc->toc_entry[i].key != key); + Assert(toc->toc_entry[i].key != key); #endif toc_bytes = offsetof(shm_toc, toc_entry) + nentry * sizeof(shm_toc_entry) @@ -208,8 +205,8 @@ shm_toc_insert(shm_toc *toc, uint64 key, void *address) } Assert(offset < total_bytes); - vtoc->toc_entry[nentry].key = key; - vtoc->toc_entry[nentry].offset = offset; + toc->toc_entry[nentry].key = key; + toc->toc_entry[nentry].offset = offset; /* * By placing a write barrier after filling in the entry and before @@ -218,7 +215,7 @@ shm_toc_insert(shm_toc *toc, uint64 key, void *address) */ pg_write_barrier(); - vtoc->toc_nentry++; + toc->toc_nentry++; SpinLockRelease(&toc->toc_mutex); } diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c index 8d246ed5a4e..5ee80c7632e 100644 --- a/src/backend/storage/lmgr/lock.c +++ b/src/backend/storage/lmgr/lock.c @@ -312,7 +312,7 @@ typedef struct uint32 count[FAST_PATH_STRONG_LOCK_HASH_PARTITIONS]; } FastPathStrongRelationLockData; -static volatile FastPathStrongRelationLockData *FastPathStrongRelationLocks; +static FastPathStrongRelationLockData *FastPathStrongRelationLocks; static void LockManagerShmemRequest(void *arg); static void LockManagerShmemInit(void *arg); diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c index 7d01c981a1f..87dd289f626 100644 --- a/src/backend/storage/lmgr/proc.c +++ b/src/backend/storage/lmgr/proc.c @@ -660,8 +660,7 @@ InitAuxiliaryProcess(void) } /* Mark auxiliary proc as in use by me */ - /* use volatile pointer to prevent code rearrangement */ - ((volatile PGPROC *) auxproc)->pid = MyProcPid; + auxproc->pid = MyProcPid; SpinLockRelease(&ProcGlobal->freeProcsLock); diff --git a/src/test/modules/test_shm_mq/setup.c b/src/test/modules/test_shm_mq/setup.c index 4f40a61e3d9..991fe27a7fa 100644 --- a/src/test/modules/test_shm_mq/setup.c +++ b/src/test/modules/test_shm_mq/setup.c @@ -38,7 +38,7 @@ static worker_state *setup_background_workers(int nworkers, dsm_segment *seg); static void cleanup_background_workers(dsm_segment *seg, Datum arg); static void wait_for_workers_to_become_ready(worker_state *wstate, - volatile test_shm_mq_header *hdr); + test_shm_mq_header *hdr); static bool check_worker_status(worker_state *wstate); /* value cached, fetched from shared memory */ @@ -258,7 +258,7 @@ cleanup_background_workers(dsm_segment *seg, Datum arg) static void wait_for_workers_to_become_ready(worker_state *wstate, - volatile test_shm_mq_header *hdr) + test_shm_mq_header *hdr) { bool result = false; diff --git a/src/test/modules/test_shm_mq/worker.c b/src/test/modules/test_shm_mq/worker.c index e13c05ae5c7..0ba1cfcac47 100644 --- a/src/test/modules/test_shm_mq/worker.c +++ b/src/test/modules/test_shm_mq/worker.c @@ -52,7 +52,7 @@ test_shm_mq_main(Datum main_arg) shm_toc *toc; shm_mq_handle *inqh; shm_mq_handle *outqh; - volatile test_shm_mq_header *hdr; + test_shm_mq_header *hdr; int myworkernumber; PGPROC *registrant; -- 2.50.1 (Apple Git-155) --hE8tq9meMVMcgUcn-- ^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2026-06-30 21:41 UTC | newest] Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-01-22 17:33 [PATCH v8 1/3] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-01-22 17:33 [PATCH v9 1/3] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-01-22 17:33 [PATCH v10 1/3] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-02-06 15:26 [PATCH v13 1/5] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-02-06 15:26 [PATCH v11 1/4] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-02-06 15:26 [PATCH v11 1/4] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-02-06 15:26 [PATCH v12 1/4] Remove some unnecessary optimizations in popcount code. Nathan Bossart <nathan@postgresql.org> 2026-06-30 21:41 [PATCH v1 1/1] Remove unnecessary volatile qualifiers. Nathan Bossart <nathan@postgresql.org>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox