public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v6 7/7] Update comment obsolete since 69c3936a
36+ messages / 6 participants
[nested] [flat]
* [PATCH v6 7/7] Update comment obsolete since 69c3936a
@ 2020-02-15 21:53 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Justin Pryzby @ 2020-02-15 21:53 UTC (permalink / raw)
---
src/backend/executor/nodeAgg.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAgg.c b/src/backend/executor/nodeAgg.c
index 3d60f7a314..567d65bb6c 100644
--- a/src/backend/executor/nodeAgg.c
+++ b/src/backend/executor/nodeAgg.c
@@ -1538,8 +1538,7 @@ lookup_hash_entry(AggState *aggstate, uint32 hash)
}
/*
- * Look up hash entries for the current tuple in all hashed grouping sets,
- * returning an array of pergroup pointers suitable for advance_aggregates.
+ * Look up hash entries for the current tuple in all hashed grouping sets.
*
* Be aware that lookup_hash_entry can reset the tmpcontext.
*/
--
2.17.0
--xHbokkKX1kTiQeDC--
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v8 8/8] Update comment obsolete since 69c3936a
@ 2020-02-15 21:53 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Justin Pryzby @ 2020-02-15 21:53 UTC (permalink / raw)
---
src/backend/executor/nodeAgg.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAgg.c b/src/backend/executor/nodeAgg.c
index 4cd09e2291..68fe4d3591 100644
--- a/src/backend/executor/nodeAgg.c
+++ b/src/backend/executor/nodeAgg.c
@@ -2025,8 +2025,7 @@ lookup_hash_entry(AggState *aggstate, uint32 hash)
}
/*
- * Look up hash entries for the current tuple in all hashed grouping sets,
- * returning an array of pergroup pointers suitable for advance_aggregates.
+ * Look up hash entries for the current tuple in all hashed grouping sets.
*
* Be aware that lookup_hash_entry can reset the tmpcontext.
*
--
2.17.0
--ZljC5FVPx7rxDQQ8--
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v7 7/9] Update comment obsolete since 69c3936a
@ 2020-02-15 21:53 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Justin Pryzby @ 2020-02-15 21:53 UTC (permalink / raw)
---
src/backend/executor/nodeAgg.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/src/backend/executor/nodeAgg.c b/src/backend/executor/nodeAgg.c
index 867da6eebf..9aa2941457 100644
--- a/src/backend/executor/nodeAgg.c
+++ b/src/backend/executor/nodeAgg.c
@@ -1538,8 +1538,7 @@ lookup_hash_entry(AggState *aggstate, uint32 hash)
}
/*
- * Look up hash entries for the current tuple in all hashed grouping sets,
- * returning an array of pergroup pointers suitable for advance_aggregates.
+ * Look up hash entries for the current tuple in all hashed grouping sets.
*
* Be aware that lookup_hash_entry can reset the tmpcontext.
*/
--
2.17.0
--UfEAyuTBtIjiZzX6
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment; filename="v7-0008-Add-explain-MACHINE.patch"
^ permalink raw reply [nested|flat] 36+ messages in thread
* [PATCH v5 05/14] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_//gBZFZYGWm7urqUgDbu6PSe
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v5-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-16 01:27 Tom Lane <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Tom Lane @ 2025-01-16 01:27 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: John Naylor <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
David Rowley <[email protected]> writes:
> I agree that the evidence you (John) gathered is enough reason to use memcpy().
Okay ... doesn't quite match my intuition, but intuition is a poor
guide to such things.
regards, tom lane
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-17 17:15 Nathan Bossart <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-01-17 17:15 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: David Rowley <[email protected]>; John Naylor <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
With commit e24d770 in place, I took a closer look at hex_decode(), and I
concluded that doing anything better without intrinsics would likely
require either a huge lookup table or something with complexity rivalling
the instrinsics approach (while also not rivalling its performance). So, I
took a closer look at the instrinsics patches and had the following
thoughts:
* The approach looks generally reasonable to me, but IMHO the code needs
much more commentary to explain how it works.
* The functions that test the length before potentially calling a function
pointer should probably be inlined (see pg_popcount() in pg_bitutils.h).
I wouldn't be surprised if some compilers are inlining this stuff
already, but it's probably worth being explicit about it.
* Finally, I think we should ensure we've established a really strong case
for this optimization. IME these intrinsics patches require a ton of
time and energy, and the code is often extremely complex. I would be
interested to see how your bytea test compares with the improvements
added in commit e24d770 and with sending the data in binary.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-22 10:58 [email protected] <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: [email protected] @ 2025-01-22 10:58 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> The approach looks generally reasonable to me, but IMHO the code needs
much more commentary to explain how it works.
Added comments to explain the SVE implementation.
> I would be interested to see how your bytea test compares with the
improvements added in commit e24d770 and with sending the data in binary.
The following are the bytea test results with commit e24d770.
The same query and tables were used.
With commit e24d770:
Query exec time: 2.324 sec
hex_encode function time: 0.72 sec
Pre-commit e24d770:
Query exec time: 2.858 sec
hex_encode function time: 1.228 sec
SVE patch:
Query exec time: 1.654 sec
hex_encode_sve function time: 0.085 sec
> The functions that test the length before potentially calling a function
> pointer should probably be inlined (see pg_popcount() in pg_bitutils.h).
> I wouldn't be surprised if some compilers are inlining this stuff
> already, but it's probably worth being explicit about it.
Should we implement an inline function in "utils/builtins.h", similar to
pg_popcount()? Currently, we have not modified the header file, everything
is statically implemented in encode.c.
---
Chiranmoy
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-22 11:10 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-01-22 11:10 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
I realized I didn't attach the patch.
Attachments:
[application/octet-stream] v2-0001-SVE-support-for-hex-encode-and-hex-decode.patch (15.8K, ../../OSBPR01MB266437C48E1F5ACB3D8C69C197E12@OSBPR01MB2664.jpnprd01.prod.outlook.com/3-v2-0001-SVE-support-for-hex-encode-and-hex-decode.patch)
download | inline diff:
From 2094bc7f60db93693f2c054e9044d8baa128bb8f Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Wed, 22 Jan 2025 15:52:40 +0530
Subject: [PATCH v2] SVE support for hex encode and hex decode
---
config/c-compiler.m4 | 53 ++++++++
configure | 63 +++++++++
configure.ac | 9 ++
meson.build | 47 +++++++
src/backend/utils/adt/encode.c | 241 +++++++++++++++++++++++++++++++++
src/include/pg_config.h.in | 3 +
6 files changed, 416 insertions(+)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 8534cc54c1..bb22ceed17 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -704,3 +704,56 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_AVX512_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the ARM SVE intrinsic required for hex coding:
+# svld1, svtbl, svsel, etc.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[
+ AC_CACHE_CHECK([for svtbl, svlsr_z, svand_z, svcreate2, svst2, svsel and svget2 intrinsics],
+ [pgac_cv_arm_sve_hex_intrinsics],
+ [
+
+ AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>],
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+
+ [
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_z(pred, bytes, 4);
+ bytes = svand_z(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, bytes);
+ cmp1 = svcmplt(pred, bytes, 0);
+ cmp2 = svcmpgt(pred, bytes, 0);
+ bytes = svsel(svnot_z(pred, svand_z(pred, cmp1, cmp2)), bytes, bytes);
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ ])],
+ [pgac_cv_arm_sve_hex_intrinsics=yes],
+ [pgac_cv_arm_sve_hex_intrinsics=no])
+
+ ])
+
+ if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics = yes
+ fi
+])
diff --git a/configure b/configure
index ceeef9b091..e634feec02 100755
--- a/configure
+++ b/configure
@@ -17168,6 +17168,69 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for SVE intrinsic svtbl, svlsr_z, etc." >&5
+ $as_echo_n "checking for SVE intrinsic svtbl, svlsr_z... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute(target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int
+main ()
+{
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_z(pred, bytes, 4);
+ bytes = svand_z(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, bytes);
+ cmp1 = svcmplt(pred, bytes, 0);
+ cmp2 = svcmpgt(pred, bytes, 0);
+ bytes = svsel(svnot_z(pred, svand_z(pred, cmp1, cmp2)), bytes, bytes);
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS=yes
+fi
+
+if test x"$PGAC_ARM_SVE_HEX_INTRINSICS" = x"yes"; then
+ $as_echo "#define USE_SVE_WITH_RUNTIME_CHECK 1" >>confdefs.h
+fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index d713360f34..cc805667b9 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2021,6 +2021,15 @@ if test x"$host_cpu" = x"x86_64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$PGAC_ARM_SVE_HEX_INTRINSICS" = x"yes"; then
+ AC_DEFINE(USE_SVE_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index 32fc89f3a4..d9d13b3c55 100644
--- a/meson.build
+++ b/meson.build
@@ -2194,6 +2194,53 @@ int main(void)
endif
+###############################################################
+# Check the availability of ARM SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_z(pred, bytes, 4);
+ bytes = svand_z(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, bytes);
+ cmp1 = svcmplt(pred, bytes, 0);
+ cmp2 = svcmpgt(pred, bytes, 0);
+ bytes = svsel(svnot_z(pred, svand_z(pred, cmp1, cmp2)), bytes, bytes);
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+}
+'''
+
+ if cc.links(prog, name: 'ARM SVE hex encoding', args: test_c_args)
+ cdata.set('USE_SVE_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d..0fe41a8d00 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -20,6 +20,10 @@
#include "utils/memutils.h"
#include "varatt.h"
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+#include <sys/auxv.h>
+#include <arm_sve.h>
+#endif
/*
* Encoding conversion API.
@@ -177,8 +181,106 @@ static const int8 hexlookup[128] = {
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
};
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+static uint64 hex_encode_slow(const char *src, size_t len, char *dst);
+static uint64 hex_decode_slow(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_slow(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst,
+ Node *escontext);
+uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext) =
+ hex_decode_safe_choose;
+
+/*
+ * Returns true if the CPU supports SVE instructions.
+ */
+static inline bool
+check_sve_support(void)
+{
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_slow;
+ hex_decode_optimized = hex_decode_slow;
+ hex_decode_safe_optimized = hex_decode_safe_slow;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+ if (len < 16)
+ return hex_encode_slow(src, len, dst);
+ return hex_encode_optimized(src, len, dst);
+}
+
+uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+ if (len < 32)
+ return hex_decode_slow(src, len, dst);
+ return hex_decode_optimized(src, len, dst);
+}
+
+uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+ if (len < 32)
+ return hex_decode_safe_slow(src, len, dst, escontext);
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+uint64
+hex_encode_slow(const char *src, size_t len, char *dst)
+#else
uint64
hex_encode(const char *src, size_t len, char *dst)
+#endif
{
const char *end = src + len;
@@ -207,14 +309,24 @@ get_hex(const char *cp, char *out)
return (res >= 0);
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+uint64
+hex_decode_slow(const char *src, size_t len, char *dst)
+#else
uint64
hex_decode(const char *src, size_t len, char *dst)
+#endif
{
return hex_decode_safe(src, len, dst, NULL);
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+uint64
+hex_decode_safe_slow(const char *src, size_t len, char *dst, Node *escontext)
+#else
uint64
hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+#endif
{
const char *s,
*srcend;
@@ -254,6 +366,135 @@ hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
return p - dst;
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+/*
+ * SVE implementation of hex_encode and hex_decode.
+ */
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ svbool_t pred;
+ svuint8_t bytes,
+ high,
+ low,
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svuint8x2_t merged;
+ uint32 vec_len = svcntb();
+
+ for (size_t i = 0; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8(i, len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_z(pred, bytes, 4); /* shift-right to get the high nibble */
+ low = svand_z(pred, bytes, 0xF); /* mask high to get the low nibble */
+
+ /*
+ * Convert the nibbles to hex digits by indexing into hextbl_vec,
+ * for example, a nibble value of 10 indexed into hextbl_vec gives 'a'.
+ * Finally, interleave the high and low nibbles
+ */
+ merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, merged);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII values '0'-'9' to integers 0-9 by subtracting 48.
+ * Similarly, convert letters 'A'-'F' and 'a'-'f' to integers 10-15.
+ */
+ svuint8_t dgt_vec = svsub_x(pred, vec, 48),
+ cap_vec = svsub_x(pred, vec, 55),
+ sml_vec = svsub_x(pred, vec, 87),
+ letter_vec;
+ /*
+ * Identify valid integers in dgt_vec, cap_vec, and sml_vec.
+ * Values 0-9 are valid in dgt_vec, while values 10-15 are valid
+ * in cap_vec and sml_vec.
+ */
+ svbool_t dgt_bool = svcmplt(pred, dgt_vec, 10),
+ cap_bool = svcmplt(pred, cap_vec, 16),
+ letter_bool;
+ /*
+ * Combine cap_vec and sml_vec and mark the valid range 10-15.
+ */
+ letter_vec = svsel(cap_bool, cap_vec, sml_vec);
+ letter_bool = svand_z(pred, svcmpgt(pred, letter_vec, 9),
+ svcmplt(pred, letter_vec, 16));
+ /*
+ * Check for invalid hexadecimal digits. Each value must fall
+ * within the range 0-9 (true in dgt_bool) or 10-15 (true in letter_bool).
+ */
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, dgt_bool, letter_bool))))
+ return false;
+
+ /* Finally, combine dgt_vec and letter_vec */
+ *res = svsel(dgt_bool, dgt_vec, letter_vec);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ svbool_t pred;
+ svuint8x2_t bytes;
+ svuint8_t high,
+ low;
+ uint32 processed;
+ size_t i = 0,
+ loop_bytes = len & ~1; /* handles inputs of odd length */
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ pred = svwhilelt_b8(i / 2, len / 2);
+ bytes = svld2(pred, (uint8 *) src);
+ high = svget2(bytes, 0); /* hex digit for high nibble */
+ low = svget2(bytes, 1); /* hex digit for low nibble */
+
+ /* fall back if ASCII less than '0' is found */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'),
+ svcmplt(pred, low, '0'))))
+ break;
+
+ /* fall back if invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* left-shift high and perform bitwise OR with low to form the byte */
+ svst1(pred, (uint8 *) dst, svorr_x(pred, svlsl_x(pred, high, 4), low));
+
+ processed = svcntp_b8(pred, pred) * 2;
+ src += processed;
+ i += processed;
+ dst += processed / 2;
+ }
+
+ if (i < len) /* fall back */
+ return dst - p + hex_decode_safe_slow(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
static uint64
hex_enc_len(const char *src, size_t srclen)
{
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798ab..b5096c11f4 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -648,6 +648,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-25 22:07 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Nathan Bossart @ 2025-01-25 22:07 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Jan 22, 2025 at 10:58:09AM +0000, [email protected] wrote:
>> The functions that test the length before potentially calling a function
>> pointer should probably be inlined (see pg_popcount() in pg_bitutils.h).
>> I wouldn't be surprised if some compilers are inlining this stuff
>> already, but it's probably worth being explicit about it.
>
> Should we implement an inline function in "utils/builtins.h", similar to
> pg_popcount()? Currently, we have not modified the header file, everything
> is statically implemented in encode.c.
Yeah, that's what I'm currently thinking we should do.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-25 22:07 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-01-25 22:07 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Jan 22, 2025 at 11:10:10AM +0000, [email protected] wrote:
> I realized I didn't attach the patch.
Thanks. Would you mind creating a commitfest entry for this one?
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-02-04 06:06 [email protected] <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-02-04 06:06 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Inlined the hex encode/decode functions in "src/include/utils/builtins.h"
similar to pg_popcount() in pg_bitutils.h.
---
Chiranmoy
Attachments:
[application/octet-stream] v3-0001-SVE-support-for-hex-encode-and-hex-decode.patch (18.3K, ../../TY2PR01MB26671195A71369185F7BD08C97F42@TY2PR01MB2667.jpnprd01.prod.outlook.com/3-v3-0001-SVE-support-for-hex-encode-and-hex-decode.patch)
download | inline diff:
From 015afdfa5b1eccc039bc1c276dd7a51d3729257a Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Tue, 4 Feb 2025 11:26:41 +0530
Subject: [PATCH v3] SVE support for hex encode and hex decode
---
config/c-compiler.m4 | 58 +++++++++
configure | 79 ++++++++++++
configure.ac | 9 ++
meson.build | 56 +++++++++
src/backend/utils/adt/encode.c | 212 ++++++++++++++++++++++++++++++++-
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 55 ++++++++-
7 files changed, 466 insertions(+), 6 deletions(-)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 8534cc54c1..d99ecfb2a7 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -704,3 +704,61 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_AVX512_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the ARM SVE intrinsic required for hex coding:
+# svtbl, svlsr_x, svand_z, svcreate2, etc.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input@<:@32@:>@;
+ char output@<:@32@:>@;
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@ == 0;
+ }
+
+ return 0;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index ceeef9b091..e445cb1451 100755
--- a/configure
+++ b/configure
@@ -17168,6 +17168,85 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_z, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_z, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input[32];
+ char output[32];
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index d713360f34..2dbb678cae 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2021,6 +2021,15 @@ if test x"$host_cpu" = x"x86_64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index 8e128f4982..6a10331acf 100644
--- a/meson.build
+++ b/meson.build
@@ -2194,6 +2194,62 @@ int main(void)
endif
+###############################################################
+# Check the availability of ARM SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int vec_len = svcntb();
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+}
+'''
+
+ if cc.links(prog, name: 'ARM SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d..cf0137a1f1 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -20,6 +20,12 @@
#include "utils/memutils.h"
#include "varatt.h"
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+#endif
/*
* Encoding conversion API.
@@ -177,8 +183,81 @@ static const int8 hexlookup[128] = {
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
};
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst,
+ Node *escontext);
+uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext) =
+ hex_decode_safe_choose;
+
+/*
+ * Returns true if the CPU supports SVE instructions.
+ */
+static inline bool
+check_sve_support(void)
+{
+#if defined(HAVE_ELF_AUX_INFO) && defined(__aarch64__) /* FreeBSD */
+ unsigned long value;
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL) && defined(__aarch64__) /* Linux */
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +287,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
@@ -254,6 +333,133 @@ hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
return p - dst;
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+/*
+ * SVE implementation of hex_encode and hex_decode.
+ */
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ svbool_t pred;
+ svuint8_t bytes,
+ high,
+ low,
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svuint8x2_t merged;
+ uint32 vec_len = svcntb();
+
+ for (size_t i = 0; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8(i, len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4); /* shift-right to get the high nibble */
+ low = svand_z(pred, bytes, 0xF); /* mask high to get the low nibble */
+
+ /*
+ * Convert the nibbles to hex digits by indexing into hextbl_vec,
+ * for example, a nibble value of 10 indexed into hextbl_vec gives 'a'.
+ * Finally, interleave the high and low nibbles.
+ */
+ merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, merged);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII values '0'-'9' to integers 0-9 by subtracting 48.
+ * Similarly, convert letters 'A'-'F' and 'a'-'f' to integers 10-15.
+ */
+ svuint8_t dgt_vec = svsub_x(pred, vec, 48),
+ cap_vec = svsub_x(pred, vec, 55),
+ sml_vec = svsub_x(pred, vec, 87),
+ ltr_vec;
+ /*
+ * Identify valid integers in dgt_vec, cap_vec, and sml_vec.
+ * Integers 0-9 are valid in dgt_vec, while integers 10-15 are valid
+ * in cap_vec and sml_vec.
+ */
+ svbool_t valid_dgt = svcmplt(pred, dgt_vec, 10),
+ valid_ltr;
+
+ /* Combine cap_vec and sml_vec and mark the valid range 10-15. */
+ ltr_vec = svsel(svcmplt(pred, cap_vec, 16), cap_vec, sml_vec);
+ valid_ltr = svand_z(pred, svcmpgt(pred, ltr_vec, 9),
+ svcmplt(pred, ltr_vec, 16));
+ /*
+ * Check for invalid hexadecimal digits. Each value must fall
+ * within the range 0-9 (true in valid_dgt) or 10-15 (true in valid_ltr).
+ */
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_dgt, valid_ltr))))
+ return false;
+
+ /* Finally, combine dgt_vec and ltr_vec */
+ *res = svsel(valid_dgt, dgt_vec, ltr_vec);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ svbool_t pred;
+ svuint8x2_t bytes;
+ svuint8_t high,
+ low;
+ uint32 processed;
+ size_t i = 0,
+ loop_bytes = len & ~1; /* handles inputs of odd length */
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ pred = svwhilelt_b8(i / 2, len / 2);
+ bytes = svld2(pred, (uint8 *) src);
+ high = svget2(bytes, 0); /* hex digits for high nibble */
+ low = svget2(bytes, 1); /* hex digits for low nibble */
+
+ /* fallback if a character below ASCII '0' is found. */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'),
+ svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* left-shift high and perform bitwise OR with low to form the byte */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ processed = svcntp_b8(pred, pred) * 2;
+ src += processed;
+ i += processed;
+ dst += processed / 2;
+ }
+
+ if (i < len) /* fall back */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
static uint64
hex_enc_len(const char *src, size_t srclen)
{
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798ab..b5096c11f4 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -648,6 +648,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d225..e9b1f963dd 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,60 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * We can use SVE intrinsics for hex-coding, but only if we can
+ * verify that the CPU supports it via a runtime check.
+ */
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext);
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-02-19 08:19 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-02-19 08:19 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
It seems that the patch doesn't compile on macOS, it is unable to map 'i'
and 'len' which are of type 'size_t' to 'uint64'. This appears to be a mac specific
issue. The latest patch should resolve this by casting 'size_t' to 'uint64' before
passing it to 'svwhilelt_b8'.
[11:04:07.478] ../src/backend/utils/adt/encode.c:356:10: error: call to 'svwhilelt_b8' is ambiguous
[11:04:07.478] 356 | pred = svwhilelt_b8(i, len);
[11:04:07.478] | ^~~~~~~~~~~~
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28288:10: note: candidate function
[11:04:07.478] 28288 | svbool_t svwhilelt_b8(uint32_t, uint32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28296:10: note: candidate function
[11:04:07.478] 28296 | svbool_t svwhilelt_b8(uint64_t, uint64_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28304:10: note: candidate function
[11:04:07.478] 28304 | svbool_t svwhilelt_b8(int32_t, int32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28312:10: note: candidate function
[11:04:07.478] 28312 | svbool_t svwhilelt_b8(int64_t, int64_t);
[11:04:07.478] | ^
[11:04:07.478] ../src/backend/utils/adt/encode.c:433:10: error: call to 'svwhilelt_b8' is ambiguous
[11:04:07.478] 433 | pred = svwhilelt_b8(i / 2, len / 2);
[11:04:07.478] | ^~~~~~~~~~~~
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28288:10: note: candidate function
[11:04:07.478] 28288 | svbool_t svwhilelt_b8(uint32_t, uint32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28296:10: note: candidate function
[11:04:07.478] 28296 | svbool_t svwhilelt_b8(uint64_t, uint64_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28304:10: note: candidate function
[11:04:07.478] 28304 | svbool_t svwhilelt_b8(int32_t, int32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28312:10: note: candidate function
[11:04:07.478] 28312 | svbool_t svwhilelt_b8(int64_t, int64_t);
[11:04:07.478] | ^
[11:04:07.478] 2 errors generated.
---
Chiranmoy
Attachments:
[application/octet-stream] v4-0001-SVE-support-for-hex-encode-and-hex-decode.patch (18.4K, ../../OSBPR01MB2664ED032A81FBF21BEC82EE97C52@OSBPR01MB2664.jpnprd01.prod.outlook.com/3-v4-0001-SVE-support-for-hex-encode-and-hex-decode.patch)
download | inline diff:
From 015afdfa5b1eccc039bc1c276dd7a51d3729257a Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Tue, 4 Feb 2025 11:26:41 +0530
Subject: [PATCH v3] SVE support for hex encode and hex decode
---
config/c-compiler.m4 | 58 +++++++++
configure | 79 ++++++++++++
configure.ac | 9 ++
meson.build | 56 +++++++++
src/backend/utils/adt/encode.c | 212 ++++++++++++++++++++++++++++++++-
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 55 ++++++++-
7 files changed, 466 insertions(+), 6 deletions(-)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 8534cc54c1..d99ecfb2a7 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -704,3 +704,61 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_AVX512_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the ARM SVE intrinsic required for hex coding:
+# svtbl, svlsr_x, svand_z, svcreate2, etc.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input@<:@32@:>@;
+ char output@<:@32@:>@;
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@ == 0;
+ }
+
+ return 0;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index ceeef9b091..e445cb1451 100755
--- a/configure
+++ b/configure
@@ -17168,6 +17168,85 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_z, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_z, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input[32];
+ char output[32];
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index d713360f34..2dbb678cae 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2021,6 +2021,15 @@ if test x"$host_cpu" = x"x86_64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index 8e128f4982..6a10331acf 100644
--- a/meson.build
+++ b/meson.build
@@ -2194,6 +2194,62 @@ int main(void)
endif
+###############################################################
+# Check the availability of ARM SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int vec_len = svcntb();
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+}
+'''
+
+ if cc.links(prog, name: 'ARM SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d..cf0137a1f1 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -20,6 +20,12 @@
#include "utils/memutils.h"
#include "varatt.h"
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+#endif
/*
* Encoding conversion API.
@@ -177,8 +183,81 @@ static const int8 hexlookup[128] = {
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
};
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst,
+ Node *escontext);
+uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext) =
+ hex_decode_safe_choose;
+
+/*
+ * Returns true if the CPU supports SVE instructions.
+ */
+static inline bool
+check_sve_support(void)
+{
+#if defined(HAVE_ELF_AUX_INFO) && defined(__aarch64__) /* FreeBSD */
+ unsigned long value;
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL) && defined(__aarch64__) /* Linux */
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +287,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
@@ -254,6 +333,133 @@ hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
return p - dst;
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+/*
+ * SVE implementation of hex_encode and hex_decode.
+ */
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ svbool_t pred;
+ svuint8_t bytes,
+ high,
+ low,
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svuint8x2_t merged;
+ uint32 vec_len = svcntb();
+
+ for (size_t i = 0; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8((uint64) i, (uint64) len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4); /* shift-right to get the high nibble */
+ low = svand_z(pred, bytes, 0xF); /* mask high to get the low nibble */
+
+ /*
+ * Convert the nibbles to hex digits by indexing into hextbl_vec,
+ * for example, a nibble value of 10 indexed into hextbl_vec gives 'a'.
+ * Finally, interleave the high and low nibbles.
+ */
+ merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, merged);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII values '0'-'9' to integers 0-9 by subtracting 48.
+ * Similarly, convert letters 'A'-'F' and 'a'-'f' to integers 10-15.
+ */
+ svuint8_t dgt_vec = svsub_x(pred, vec, 48),
+ cap_vec = svsub_x(pred, vec, 55),
+ sml_vec = svsub_x(pred, vec, 87),
+ ltr_vec;
+ /*
+ * Identify valid integers in dgt_vec, cap_vec, and sml_vec.
+ * Integers 0-9 are valid in dgt_vec, while integers 10-15 are valid
+ * in cap_vec and sml_vec.
+ */
+ svbool_t valid_dgt = svcmplt(pred, dgt_vec, 10),
+ valid_ltr;
+
+ /* Combine cap_vec and sml_vec and mark the valid range 10-15. */
+ ltr_vec = svsel(svcmplt(pred, cap_vec, 16), cap_vec, sml_vec);
+ valid_ltr = svand_z(pred, svcmpgt(pred, ltr_vec, 9),
+ svcmplt(pred, ltr_vec, 16));
+ /*
+ * Check for invalid hexadecimal digits. Each value must fall
+ * within the range 0-9 (true in valid_dgt) or 10-15 (true in valid_ltr).
+ */
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_dgt, valid_ltr))))
+ return false;
+
+ /* Finally, combine dgt_vec and ltr_vec */
+ *res = svsel(valid_dgt, dgt_vec, ltr_vec);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ svbool_t pred;
+ svuint8x2_t bytes;
+ svuint8_t high,
+ low;
+ uint32 processed;
+ size_t i = 0,
+ loop_bytes = len & ~1; /* handles inputs of odd length */
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ pred = svwhilelt_b8((uint64) i / 2, (uint64) len / 2);
+ bytes = svld2(pred, (uint8 *) src);
+ high = svget2(bytes, 0); /* hex digits for high nibble */
+ low = svget2(bytes, 1); /* hex digits for low nibble */
+
+ /* fallback if a character below ASCII '0' is found. */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'),
+ svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* left-shift high and perform bitwise OR with low to form the byte */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ processed = svcntp_b8(pred, pred) * 2;
+ src += processed;
+ i += processed;
+ dst += processed / 2;
+ }
+
+ if (i < len) /* fall back */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
static uint64
hex_enc_len(const char *src, size_t srclen)
{
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798ab..b5096c11f4 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -648,6 +648,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d225..e9b1f963dd 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,60 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * We can use SVE intrinsics for hex-coding, but only if we can
+ * verify that the CPU supports it via a runtime check.
+ */
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext);
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-06-04 13:47 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-06-04 13:47 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
I have marked the commitfest entry for this [0] as waiting-on-author
because the patch needs to be rebased.
[0] https://commitfest.postgresql.org/patch/5538/
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-06-09 09:49 [email protected] <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-06-09 09:49 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Here's the rebased patch with a few modifications.
The hand-unrolled hex encode performs better than the non-unrolled version on
r8g.4xlarge. No improvement on m7g.4xlarge.
Added line-by-line comments explaining the changes with an example.
Below are the results. Input size is in bytes, and exec time is in ms.
encode - r8g.4xlarge
Input | master | SVE | SVE-unrolled
-------+--------+--------+--------------
8 | 4.971 | 6.434 | 6.623
16 | 8.532 | 4.399 | 4.710
24 | 12.296 | 5.007 | 5.780
32 | 16.003 | 5.027 | 5.234
40 | 19.628 | 5.807 | 6.201
48 | 23.277 | 5.815 | 6.222
56 | 26.927 | 6.744 | 7.030
64 | 30.419 | 6.774 | 6.347
128 | 83.250 | 10.214 | 9.104
256 |112.158 | 17.892 | 16.313
512 |216.544 | 31.060 | 29.876
1024 |429.351 | 59.310 | 53.374
2048 |854.677 |116.769 | 101.004
4096 |1706.528|237.322 | 195.297
8192 |3723.884|499.520 | 385.424
---------------------------------------
encode - m7g.4xlarge
Input | master | SVE | SVE-unrolled
-------+--------+--------+--------------
8 | 5.503 | 7.986 | 8.053
16 | 9.881 | 9.583 | 9.888
24 | 13.854 | 9.212 | 10.138
32 | 18.056 | 9.208 | 9.364
40 | 22.127 | 10.134 | 10.540
48 | 26.214 | 10.186 | 10.550
56 | 29.718 | 10.197 | 10.428
64 | 33.613 | 10.982 | 10.497
128 | 66.060 | 12.460 | 12.624
256 |130.225 | 18.491 | 18.872
512 |267.105 | 30.343 | 31.661
1024 |515.603 | 54.371 | 55.341
2048 |1013.766|103.898 | 105.192
4096 |2018.705|202.653 | 203.142
8192 |4000.496|400.918 | 401.842
---------------------------------------
decode - r8g.4xlarge
Input | master | SVE
-------+--------+--------
8 | 7.641 | 8.787
16 | 14.301 | 14.477
32 | 28.663 | 6.091
48 | 42.940 | 17.604
64 | 57.483 | 10.549
80 | 71.637 | 19.194
96 | 85.918 | 15.586
112 |100.272 | 25.956
128 |114.740 | 19.829
256 |229.176 | 36.032
512 |458.295 | 68.222
1024 |916.741 |132.927
2048 |1833.422|262.741
4096 |3667.096|522.009
8192 |7333.886|1042.447
---------------------------------------
decode - m7g.4xlarge
Input | master | SVE
-------+--------+--------
8 | 8.194 | 9.433
16 | 14.397 | 15.606
32 | 26.669 | 29.006
48 | 45.971 | 48.984
64 | 58.468 | 12.388
80 | 70.820 | 22.295
96 | 84.792 | 43.470
112 | 98.992 | 54.282
128 |113.250 | 25.508
256 |218.743 | 45.165
512 |414.133 | 86.800
1024 |828.493 |174.670
2048 |1617.921|346.375
4096 |3259.159|689.391
8192 |6551.879|1376.195
--------
Chiranmoy
Attachments:
[application/octet-stream] v5-0001-SVE-support-for-hex-coding.patch (24.6K, ../../OS9PR01MB15185A8BA8E15E4AFD8B14490976BA@OS9PR01MB15185.jpnprd01.prod.outlook.com/3-v5-0001-SVE-support-for-hex-coding.patch)
download | inline diff:
From 3a508684171ae411e4e8251c717b61a8def04c1f Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Mon, 9 Jun 2025 14:16:26 +0530
Subject: [PATCH v5] SVE support for hex coding
---
config/c-compiler.m4 | 85 ++++++++
configure | 104 +++++++++
configure.ac | 9 +
meson.build | 81 +++++++
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/encode.c | 6 +-
src/backend/utils/adt/encode_aarch64.c | 278 +++++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 51 ++++-
10 files changed, 613 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/adt/encode_aarch64.c
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 5f3e1d1faf9..20e71cd8546 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -797,3 +797,88 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_SVE_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the SVE intrinsic required for hex coding:
+# svsub_x, svcmplt, svsel, svcmpgt, svtbl, svlsr_x, svand_z, svcreate2,
+# svptest_any, svnot_z, svorr_z, svcntb, svld1, svwhilelt_b8, svst2, svld2,
+# svget2, svst1 and svlsl_x.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+
+ char input@<:@64@:>@;
+ char output@<:@128@:>@;
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index 4f15347cc95..4d5d6acefb5 100755
--- a/configure
+++ b/configure
@@ -17851,6 +17851,110 @@ $as_echo "#define USE_SVE_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_x, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_x, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+
+ char input[64];
+ char output[128];
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_HEX_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index 4b8335dc613..fcae9b84616 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2107,6 +2107,15 @@ if test x"$host_cpu" = x"aarch64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_HEX_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index d142e3e408b..de2d1ebd384 100644
--- a/meson.build
+++ b/meson.build
@@ -2384,6 +2384,87 @@ int main(void)
endif
+###############################################################
+# Check the availability of SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+
+char input[64];
+char output[128];
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+}
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char hextbl[] = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+}
+'''
+
+ if cc.links(prog, name: 'SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_HEX_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 4a233b63c32..2a3ba1d4485 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -32,6 +32,7 @@ OBJS = \
dbsize.o \
domains.o \
encode.o \
+ encode_aarch64.o \
enum.o \
expandeddatum.o \
expandedrecord.o \
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d1..fa62ce3107d 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -178,7 +178,7 @@ static const int8 hexlookup[128] = {
};
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +208,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
new file mode 100644
index 00000000000..574a7550469
--- /dev/null
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -0,0 +1,278 @@
+/*-------------------------------------------------------------------------
+ *
+ * encode_aarch64.c
+ * Holds the SVE hex encode/decode implementations.
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/encode_aarch64.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <c.h>
+
+#include "utils/builtins.h"
+
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+
+/*
+ * These are the SVE implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * The function pointers are initially set to "choose" functions. These
+ * functions will first set the pointers to the right implementations (based on
+ * what the current CPU supports) and then will call the pointer to fulfill the
+ * caller's request.
+ */
+
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext);
+uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext) = hex_decode_safe_choose;
+
+static inline bool
+check_sve_support(void)
+{
+#ifdef HAVE_ELF_AUX_INFO
+ unsigned long value;
+
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL)
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint32 vec_len = svcntb();
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svbool_t pred = svptrue_b8();
+ size_t loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svuint8_t bytes, high, low;
+ svuint8x2_t zipped;
+
+ for (size_t i = 0; i < loop_bytes; i += 2 * vec_len)
+ {
+ bytes = svld1(pred, (uint8 *) src);
+
+ /* Right-shift to obtain the high nibble */
+ high = svlsr_x(pred, bytes, 4);
+
+ /* Mask the high nibble to obtain the low nibble */
+ low = svand_z(pred, bytes, 0xF);
+
+ /*
+ * Convert the high and low nibbles to hexadecimal digits using a
+ * vectorized table lookup and zip (interleave) the hexadecimal digits.
+ */
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+
+ /* unrolled */
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ /* process remaining tail bytes */
+ for (size_t i = loop_bytes; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8((uint64) i, (uint64) len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+/*
+ * get_hex_sve
+ * Returns true if the hexadecimal digits are successfully converted
+ * to nibbles and stored in 'res'; otherwise, returns false.
+ */
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII of '0'-'9' to integers 0-9 by subtracting 48 (ASCII of '0').
+ * Similarly, convert letters 'A'–'F' and 'a'–'f' to integers 10–15 by
+ * subtracting 55 ('A' - 10) and 87 ('a' - 10).
+ */
+ svuint8_t digit = svsub_x(pred, vec, '0'),
+ upper = svsub_x(pred, vec, 'A' - 10),
+ lower = svsub_x(pred, vec, 'a' - 10);
+
+ /*
+ * Identify valid values in digits, upper, and lower vectors.
+ * Values 0-9 are valid in digits, while values 10-15 are valid
+ * in upper and lower.
+ *
+ * Example:
+ * vec: '0' '9' 'A' 'F' 'a' 'f'
+ * vec (in ASCII): 48 57 65 70 97 102
+ *
+ * digit: 0 9 17 22 49 54
+ * valid_digit: 1 1 0 0 0 0
+ *
+ * upper: 249 2 10 15 42 47
+ * valid_upper: 0 1 1 1 0 0
+ *
+ * lower: 217 226 234 239 10 15
+ *
+ * Note that values 0-9 are also marked valid in valid_upper, this will be
+ * handled later.
+ */
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+
+ /*
+ * Merge upper and lower vector using the logic: take the element from
+ * upper if it's true in valid_upper else pick the element in lower
+ *
+ * Mark the valid range i.e. 10-15 in letter vector
+ *
+ * letter: 217 2 10 15 10 15
+ * valid_letter: 0 0 1 1 1 1
+ */
+
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+
+ /*
+ * Check for invalid hexadecimal digit. Each value must fall within
+ * the range 0-9 (true in valid_digit) or 10-15 (true in valid_letter) i.e.
+ * the OR of valid_digit and valid_letter should be all true.
+ */
+
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return false;
+
+ /*
+ * Finally, combine digit and letter vectors using the logic:
+ * take the element from digit if it's true in valid_digit else pick the
+ * element in letter.
+ *
+ * res: 0 9 10 15 10 15
+ */
+
+ *res = svsel(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = svcntb();
+ size_t loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svbool_t pred = svptrue_b8();
+ const char *p = dst;
+
+ for (size_t i = 0; i < loop_bytes; i += 2 * vec_len)
+ {
+ svuint8x2_t bytes = svld2(pred, (uint8 *) src);
+ svuint8_t high = svget2(bytes, 0),
+ low = svget2(bytes, 1);
+
+ /* fallback for characters with ASCII values below '0' */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if an invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* form the byte by left-shifting the high nibble and OR-ing it with the low nibble */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > loop_bytes) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - loop_bytes, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* USE_SVE_HEX_WITH_RUNTIME_CHECK */
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index 244f48f4fd7..ea88dd77390 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -21,6 +21,7 @@ backend_sources += files(
'dbsize.c',
'domains.c',
'encode.c',
+ 'encode_aarch64.c',
'enum.c',
'expandeddatum.c',
'expandedrecord.c',
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 726a7c1be1f..7a227f1875f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -675,6 +675,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_HEX_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d2255..2f72d8df9d1 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,56 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * On AArch64, we can try to use an SVE optimized hex encode/decode on some systems.
+ */
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext);
+#endif
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-07-07 10:41 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-07-07 10:41 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Attaching the rebased patch, some regression tests for SIMD hex-coding,
and a script to test bytea performance (usage info in the script).
The results obtained using the script on an m7g.4xlarge are shown below.
Read Operation
table (MB) | HEAD (ms) | SVE (ms) | improvement (%)
---------------------------------------------------
52 | 136 | 111 | 18.38
105 | 215 | 164 | 23.72
209 | 452 | 331 | 26.76
419 | 830 | 602 | 27.46
Write Operation - table size after write
table (MB) | HEAD (ms) | SVE (ms) | improvement (%)
---------------------------------------------------
52 | 1430 | 1361 | 4.82
105 | 2956 | 2816 | 4.73
The bytea write numbers are averaged over 7 runs, with the table
truncated and vacuumed after each run.
--------
Chiranmoy
Attachments:
[text/x-python] bytea_test.py (2.4K, ../../OS9PR01MB15185626CA7C6EAF82C93ABA3974FA@OS9PR01MB15185.jpnprd01.prod.outlook.com/3-bytea_test.py)
download | inline:
from time import time
from sys import argv
import psycopg2
# import psycopg2.extras
"""
Usage
Set the input_path, output_path and conn_params in the main guard below.
To create the test table
python3 bytea_test.py c
To insert a file in the input_path N times
python3 bytea_test.py w <N>
To read the data from the table and write the first file to output_path
python3 bytea_test.py r
"""
def read_binary(input_path):
with open(input_path, 'rb') as f:
return f.read()
def create_table(conn_params):
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
cursor.execute("""CREATE TABLE IF NOT EXISTS BYTEA_TABLE (data BYTEA);""",)
conn.commit()
cursor.close()
conn.close()
def save_file(binary_data, duplicate, conn_params):
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
data = psycopg2.Binary(binary_data)
# rows = [(data,) for _ in range(duplicate)]
# start = time()
# psycopg2.extras.execute_values(cursor, """INSERT INTO bytea_table (data) VALUES %s;""", rows)
# print(time()-start)
start = time()
for _ in range(duplicate):
cursor.execute("""INSERT INTO bytea_table (data) VALUES (%s);""", (data,))
print(time()-start)
conn.commit()
cursor.close()
conn.close()
print(f"file saved {duplicate} times")
def retrieve_file(output_path, conn_params):
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
start = time()
cursor.execute("""SELECT data FROM bytea_table;""")
print(time()-start)
result = cursor.fetchone()
if result:
binary_data = result[0]
with open(output_path, 'wb') as f:
f.write(binary_data)
print(f"File retrieved, saved to '{output_path}'")
else:
print(f"ERROR")
cursor.close()
conn.close()
if __name__ == '__main__':
input_path = 'INPUT/PATH/file.jpg'
output_path = 'OUTPUT/PATCH/retrieved.jpg'
conn_params = {
'dbname': '',
'user': '',
'password': '',
'host': '',
'port': 0
}
if len(argv) == 1:
print("Specify Mode")
elif argv[1].lower() == "c":
create_table(conn_params)
elif argv[1].lower() == "r":
retrieve_file(output_path, conn_params)
elif argv[1].lower() == 'w' and int(argv[2]) > 0:
save_file(read_binary(input_path), int(argv[2]), conn_params)
else:
print("Invalid Mode")
[application/octet-stream] v1-0001-hex-coding-regress-test.patch (7.4K, ../../OS9PR01MB15185626CA7C6EAF82C93ABA3974FA@OS9PR01MB15185.jpnprd01.prod.outlook.com/4-v1-0001-hex-coding-regress-test.patch)
download | inline diff:
From 9066e3296160af4e703f90f460f8e75471b6425d Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Sun, 6 Jul 2025 19:25:28 +0530
Subject: [PATCH v1] hex coding regress test
---
src/test/regress/expected/hex_coding.out | 63 ++++++++++++++++++++++++
src/test/regress/parallel_schedule | 5 ++
src/test/regress/sql/hex_coding.sql | 39 +++++++++++++++
3 files changed, 107 insertions(+)
create mode 100644 src/test/regress/expected/hex_coding.out
create mode 100644 src/test/regress/sql/hex_coding.sql
diff --git a/src/test/regress/expected/hex_coding.out b/src/test/regress/expected/hex_coding.out
new file mode 100644
index 00000000000..e6d78fa4876
--- /dev/null
+++ b/src/test/regress/expected/hex_coding.out
@@ -0,0 +1,63 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d');
+ ^
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+ encode
+----------------------------------------------------------------------------------------------------------------------------------
+ ab
+ 01ab
+ deadc0de
+ baadf00d
+ c001c0ffee
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0de
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+(8 rows)
+
+DROP TABLE BYTEA_TABLE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index a424be2a6bf..8812d80d592 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -109,6 +109,11 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson sqljson_queryfuncs sqljson_jsontable
+# ----------
+# Another group of parallel tests for hex encode/decode
+# ----------
+test: hex_coding
+
# ----------
# Another group of parallel tests
# with depends on create_misc
diff --git a/src/test/regress/sql/hex_coding.sql b/src/test/regress/sql/hex_coding.sql
new file mode 100644
index 00000000000..97c51b62e90
--- /dev/null
+++ b/src/test/regress/sql/hex_coding.sql
@@ -0,0 +1,39 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+
+DROP TABLE BYTEA_TABLE;
--
2.34.1
[application/octet-stream] v5-0001-SVE-support-for-hex-coding.patch (24.6K, ../../OS9PR01MB15185626CA7C6EAF82C93ABA3974FA@OS9PR01MB15185.jpnprd01.prod.outlook.com/5-v5-0001-SVE-support-for-hex-coding.patch)
download | inline diff:
From 5a9bc0e99f7ae102c11cc905cd6c4df6016c415d Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Sun, 6 Jul 2025 19:35:46 +0530
Subject: [PATCH v5] SVE support for hex coding
---
config/c-compiler.m4 | 85 ++++++++
configure | 104 +++++++++
configure.ac | 9 +
meson.build | 81 +++++++
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/encode.c | 6 +-
src/backend/utils/adt/encode_aarch64.c | 280 +++++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 51 ++++-
10 files changed, 615 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/adt/encode_aarch64.c
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index da40bd6a647..73d12826698 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -798,3 +798,88 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_SVE_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the SVE intrinsic required for hex coding:
+# svsub_x, svcmplt, svsel, svcmpgt, svtbl, svlsr_x, svand_z, svcreate2,
+# svptest_any, svnot_z, svorr_z, svcntb, svld1, svwhilelt_b8, svst2, svld2,
+# svget2, svst1 and svlsl_x.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+
+ char input@<:@64@:>@;
+ char output@<:@128@:>@;
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index 16ef5b58d1a..df78a5408d3 100755
--- a/configure
+++ b/configure
@@ -17851,6 +17851,110 @@ $as_echo "#define USE_SVE_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_x, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_x, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+
+ char input[64];
+ char output[128];
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_HEX_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index b3efc49c97a..ce0015bb543 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2107,6 +2107,15 @@ if test x"$host_cpu" = x"aarch64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_HEX_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index a97854a947d..68700a4bba0 100644
--- a/meson.build
+++ b/meson.build
@@ -2389,6 +2389,87 @@ int main(void)
endif
+###############################################################
+# Check the availability of SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+
+char input[64];
+char output[128];
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+}
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char hextbl[] = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+}
+'''
+
+ if cc.links(prog, name: 'SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_HEX_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index ffeacf2b819..d2fa03efe98 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -33,6 +33,7 @@ OBJS = \
dbsize.o \
domains.o \
encode.o \
+ encode_aarch64.o \
enum.o \
expandeddatum.o \
expandedrecord.o \
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d1..fa62ce3107d 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -178,7 +178,7 @@ static const int8 hexlookup[128] = {
};
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +208,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
new file mode 100644
index 00000000000..bf8157900f8
--- /dev/null
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -0,0 +1,280 @@
+/*-------------------------------------------------------------------------
+ *
+ * encode_aarch64.c
+ * Holds the SVE hex encode/decode implementations.
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/encode_aarch64.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <c.h>
+
+#include "utils/builtins.h"
+
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+
+/*
+ * These are the SVE implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * The function pointers are initially set to "choose" functions. These
+ * functions will first set the pointers to the right implementations (based on
+ * what the current CPU supports) and then will call the pointer to fulfill the
+ * caller's request.
+ */
+
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext);
+uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext) = hex_decode_safe_choose;
+
+static inline bool
+check_sve_support(void)
+{
+#ifdef HAVE_ELF_AUX_INFO
+ unsigned long value;
+
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL)
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint32 vec_len = svcntb();
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svbool_t pred = svptrue_b8();
+ size_t loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svuint8_t bytes, high, low;
+ svuint8x2_t zipped;
+
+ for (size_t i = 0; i < loop_bytes; i += 2 * vec_len)
+ {
+ bytes = svld1(pred, (uint8 *) src);
+
+ /* Right-shift to obtain the high nibble */
+ high = svlsr_x(pred, bytes, 4);
+
+ /* Mask the high nibble to obtain the low nibble */
+ low = svand_z(pred, bytes, 0xF);
+
+ /*
+ * Convert the high and low nibbles to hexadecimal digits using a
+ * vectorized table lookup and zip (interleave) the hexadecimal digits.
+ */
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+
+ /* unrolled */
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ /* process remaining tail bytes */
+ for (size_t i = loop_bytes; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8((uint64) i, (uint64) len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+/*
+ * get_hex_sve
+ * Returns true if the hexadecimal digits are successfully converted
+ * to nibbles and stored in 'res'; otherwise, returns false.
+ */
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII of '0'-'9' to integers 0-9 by subtracting 48 (ASCII of '0').
+ * Similarly, convert letters 'A'–'F' and 'a'–'f' to integers 10–15 by
+ * subtracting 55 ('A' - 10) and 87 ('a' - 10).
+ */
+ svuint8_t digit = svsub_x(pred, vec, '0'),
+ upper = svsub_x(pred, vec, 'A' - 10),
+ lower = svsub_x(pred, vec, 'a' - 10);
+
+ /*
+ * Identify valid values in digits, upper, and lower vectors.
+ * Values 0-9 are valid in digits, while values 10-15 are valid
+ * in upper and lower.
+ *
+ * Example:
+ * vec: '0' '9' 'A' 'F' 'a' 'f'
+ * vec (in ASCII): 48 57 65 70 97 102
+ *
+ * digit: 0 9 17 22 49 54
+ * valid_digit: 1 1 0 0 0 0
+ *
+ * upper: 249 2 10 15 42 47
+ * valid_upper: 0 1 1 1 0 0
+ *
+ * lower: 217 226 234 239 10 15
+ *
+ * Note that values 0-9 are also marked valid in valid_upper, this will be
+ * handled later.
+ */
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+
+ /*
+ * Merge upper and lower vector using the logic: take the element from
+ * upper if it's true in valid_upper else pick the element in lower
+ *
+ * Mark the valid range i.e. 10-15 in letter vector
+ *
+ * letter: 217 2 10 15 10 15
+ * valid_letter: 0 0 1 1 1 1
+ */
+
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+
+ /*
+ * Check for invalid hexadecimal digit. Each value must fall within
+ * the range 0-9 (true in valid_digit) or 10-15 (true in valid_letter) i.e.
+ * the OR of valid_digit and valid_letter should be all true.
+ */
+
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return false;
+
+ /*
+ * Finally, combine digit and letter vectors using the logic:
+ * take the element from digit if it's true in valid_digit else pick the
+ * element in letter.
+ *
+ * res: 0 9 10 15 10 15
+ */
+
+ *res = svsel(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = svcntb();
+ size_t i = 0,
+ loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svbool_t pred = svptrue_b8();
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ svuint8x2_t bytes = svld2(pred, (uint8 *) src);
+ svuint8_t high = svget2(bytes, 0),
+ low = svget2(bytes, 1);
+
+ /* fallback for characters with ASCII values below '0' */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if an invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* form the byte by left-shifting the high nibble and OR-ing it with the low nibble */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ i += 2 * vec_len;
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > i) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* USE_SVE_HEX_WITH_RUNTIME_CHECK */
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index ed9bbd7b926..094a9c7c013 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -22,6 +22,7 @@ backend_sources += files(
'dbsize.c',
'domains.c',
'encode.c',
+ 'encode_aarch64.c',
'enum.c',
'expandeddatum.c',
'expandedrecord.c',
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 726a7c1be1f..7a227f1875f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -675,6 +675,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_HEX_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d2255..2f72d8df9d1 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,56 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * On AArch64, we can try to use an SVE optimized hex encode/decode on some systems.
+ */
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext);
+#endif
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-03 11:11 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 2 replies; 36+ messages in thread
From: [email protected] @ 2025-09-03 11:11 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi all,
Since the CommitFest is underway, could we get some feedback to improve the patch?
_______
Chiranmoy
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-03 14:48 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 0 replies; 36+ messages in thread
From: Nathan Bossart @ 2025-09-03 14:48 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 03, 2025 at 11:11:24AM +0000, [email protected] wrote:
> Since the CommitFest is underway, could we get some feedback to improve
> the patch?
I see that there was some discussion about a Neon implementation upthread,
but I'm not sure we concluded anything. For popcount, we first added a
Neon version before adding the SVE version, which required more complicated
configure/runtime checks. Presumably Neon is available on more hardware
than SVE, so that could be a good place to start here, too.
Also, I'd strongly encourage you to get involved with others' patches on
the mailing lists (e.g., reviewing, testing). Patch submissions are great,
but this community depends on other types of participation, too. IME
helping others with their patches also tends to incentivize others to help
with yours.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-04 06:20 John Naylor <[email protected]>
parent: [email protected] <[email protected]>
1 sibling, 1 reply; 36+ messages in thread
From: John Naylor @ 2025-09-04 06:20 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 3, 2025 at 6:11 PM [email protected]
<[email protected]> wrote:
>
> Hi all,
>
> Since the CommitFest is underway, could we get some feedback to improve the patch?
On that note, I was hoping you could give us feedback on whether the
improvement in PG18 made any difference at all in your real-world
use-case, i.e. not just in a microbenchmark, but also including
transmission of the hex-encoded values across the network to the
client (that I assume must decode them again).
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-04 14:55 [email protected] <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-09-04 14:55 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> I see that there was some discussion about a Neon implementation upthread,
> but I'm not sure we concluded anything. For popcount, we first added a
> Neon version before adding the SVE version, which required more complicated
> configure/runtime checks. Presumably Neon is available on more hardware
> than SVE, so that could be a good place to start here, too.
We have added the Neon versions of hex encode/decode.
Here are the microbenchmark numbers.
hex_encode - m7g.4xlarge
Input | Head | Neon
-------+--------+--------
32 | 18.056 | 5.957
40 | 22.127 | 10.205
48 | 26.214 | 14.151
64 | 33.613 | 6.164
128 | 66.060 | 11.372
256 |130.225 | 18.543
512 |267.105 | 33.977
1024 |515.603 | 64.462
hex_decode - m7g.4xlarge
Input | Head | Neon
-------+--------+--------
32 | 26.669 | 9.462
40 | 36.320 | 19.347
48 | 45.971 | 19.099
64 | 58.468 | 17.648
128 |113.250 | 30.437
256 |218.743 | 56.824
512 |414.133 |107.212
1024 |828.493 |210.740
> Also, I'd strongly encourage you to get involved with others' patches on
> the mailing lists (e.g., reviewing, testing). Patch submissions are great,
> but this community depends on other types of participation, too. IME
> helping others with their patches also tends to incentivize others to help
> with yours.
Sure, we will try to test/review patches on areas we have experience.
> On that note, I was hoping you could give us feedback on whether the
> improvement in PG18 made any difference at all in your real-world
> use-case, i.e. not just in a microbenchmark, but also including
> transmission of the hex-encoded values across the network to the
> client (that I assume must decode them again).
Yes, the improvement in v18 did help, check the attached perf graphs.
We used a python script to send and receive binary data from postgres.
For simple select queries on a bytea column, hex_encode was taking
42% of the query execution time in v17, this was reduced to 33% in v18,
resulting in around 18% improvement in overall query time.
The proposed patch further reduces the hex_encode function usage to
5.6%, another 25% improvement in total query time.
We observed similar improvements for insert queries on the bytea column.
hex_decode usage decreased from 15.5% to 5.5%, a 5-8% query level
improvement depending on which storage type is used.
------
Chiranmoy
Attachments:
[application/octet-stream] v6-0001-NEON-support-for-hex-coding.patch (10.2K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/3-v6-0001-NEON-support-for-hex-coding.patch)
download | inline diff:
From e642b6d32d4715c988b6b93d57385a7c0779182d Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Thu, 4 Sep 2025 12:23:24 +0530
Subject: [PATCH v6 1/3] NEON support for hex coding
---
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/encode.c | 6 +-
src/backend/utils/adt/encode_aarch64.c | 195 +++++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/include/utils/builtins.h | 57 +++++++-
5 files changed, 254 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/adt/encode_aarch64.c
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index cc68ac545a5..40eaee14899 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -33,6 +33,7 @@ OBJS = \
dbsize.o \
domains.o \
encode.o \
+ encode_aarch64.o \
enum.o \
expandeddatum.o \
expandedrecord.o \
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d1..fa62ce3107d 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -178,7 +178,7 @@ static const int8 hexlookup[128] = {
};
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +208,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
new file mode 100644
index 00000000000..7b0412dc255
--- /dev/null
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -0,0 +1,195 @@
+/*-------------------------------------------------------------------------
+ *
+ * encode_aarch64.c
+ * Holds the AArch64 hex encode/decode implementations.
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/encode_aarch64.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <c.h>
+
+#include "utils/builtins.h"
+
+#ifdef HEX_CODING_AARCH64
+
+#include <arm_neon.h>
+
+uint64
+hex_encode_optimized(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint8x16_t hextbl_vec = vld1q_u8((uint8 *) hextbl);
+ uint8x16x2_t zipped;
+ uint32 vec_len = sizeof(uint8x16_t);
+ size_t chunks_len = len & ~(2 * vec_len - 1);
+
+ for (size_t i = 0; i < chunks_len; i += 2 * vec_len)
+ {
+ uint8x16_t bytes = vld1q_u8((uint8 *) src);
+
+ /* Right-shift by 4 to get the high nibble */
+ uint8x16_t high = vshrq_n_u8(bytes, 4);
+
+ /* Mask the high nibble to get the low nibble */
+ uint8x16_t low = vandq_u8(bytes, vdupq_n_u8(0xF));
+
+ /*
+ * Convert the high and low nibbles to hexadecimal digits using a table
+ * lookup, and then zip (interleave) the resulting digits.
+ */
+ zipped.val[0] = vqtbl1q_u8(hextbl_vec, high);
+ zipped.val[1] = vqtbl1q_u8(hextbl_vec, low);
+ vst2q_u8((uint8 *) dst, zipped);
+
+ src += vec_len;
+ dst += 2 * vec_len;
+
+ /* unrolled */
+ bytes = vld1q_u8((uint8 *) src);
+ high = vshrq_n_u8(bytes, 4);
+ low = vandq_u8(bytes, vdupq_n_u8(0xF));
+
+ zipped.val[0] = vqtbl1q_u8(hextbl_vec, high);
+ zipped.val[1] = vqtbl1q_u8(hextbl_vec, low);
+ vst2q_u8((uint8 *) dst, zipped);
+
+ src += vec_len;
+ dst += 2 * vec_len;
+ }
+
+
+
+ if (len > chunks_len)
+ hex_encode_scalar(src, len - chunks_len, dst);
+
+ return (uint64) len * 2;
+}
+
+/*
+ * get_hex_neon
+ * Returns true if the hexadecimal digits are successfully converted
+ * to nibbles and stored in 'res'; otherwise, returns false.
+ */
+static inline bool
+get_hex_neon(uint8x16_t vec, uint8x16_t *res)
+{
+ /*
+ * Convert ASCII of '0'-'9' to integers 0-9 by subtracting 48 (ASCII of '0').
+ * Similarly, convert letters 'A'–'F' and 'a'–'f' to integers 10–15 by
+ * subtracting 55 ('A' - 10) and 87 ('a' - 10).
+ */
+ uint8x16_t digit = vsubq_u8(vec, vdupq_n_u8('0'));
+ uint8x16_t upper = vsubq_u8(vec, vdupq_n_u8('A' - 10));
+ uint8x16_t lower = vsubq_u8(vec, vdupq_n_u8('a' - 10));
+
+ /*
+ * Identify valid values in digit, upper, and lower vectors.
+ * Values 0-9 are valid in digits, while values 10-15 are valid
+ * in upper and lower.
+ *
+ * Example:
+ * vec: '0' '9' 'A' 'F' 'a' 'f'
+ * vec (in ASCII): 48 57 65 70 97 102
+ *
+ * digit: 0 9 17 22 49 54
+ * valid_digit: 1 1 0 0 0 0
+ *
+ * upper: 249 2 10 15 42 47
+ * valid_upper: 0 1 1 1 0 0
+ *
+ * lower: 217 226 234 239 10 15
+ *
+ * Note that values 0-9 are also marked valid in valid_upper, this will be
+ * handled later.
+ */
+
+ uint8x16_t valid_digit = vcltq_u8(digit, vdupq_n_u8(10));
+ uint8x16_t valid_upper = vcltq_u8(upper, vdupq_n_u8(16));
+
+ /*
+ * Merge upper and lower vector using the logic: pick the element from
+ * upper if it's true in valid_upper else pick the element in lower
+ *
+ * Mark the valid range i.e. 10-15 in letter vector
+ *
+ * letter: 217 2 10 15 10 15
+ * valid_letter: 0 0 1 1 1 1
+ */
+
+ uint8x16_t letter = vbslq_u8(valid_upper, upper, lower);
+ uint8x16_t valid_letter = vandq_u8(vcgtq_u8(letter, vdupq_n_u8(9)),
+ vcltq_u8(letter, vdupq_n_u8(16)));
+
+ /*
+ * Check for invalid hexadecimal digit. Each value must fall within
+ * the range 0-9 (true in valid_digit) or 10-15 (true in valid_letter) i.e.
+ * the OR of valid_digit and valid_letter should be all true.
+ */
+ uint8x16_t invalid_mask = vmvnq_u8(vorrq_u8(valid_digit, valid_letter));
+
+ if (vmaxvq_u8(invalid_mask) != 0)
+ return false;
+
+ /*
+ * Finally, combine digit and letter vectors using the logic:
+ * pick the element from digit if it's true in valid_digit else pick the
+ * element in letter.
+ *
+ * res: 0 9 10 15 10 15
+ */
+ *res = vbslq_u8(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_optimized(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_optimized(src, len, dst, NULL);
+}
+
+uint64
+hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = sizeof(uint8x16_t);
+ size_t i = 0;
+ size_t chunks_len = len & ~(2 * vec_len - 1); /* process 2 x vec_len per iteration */
+ uint8x16_t ascii_zero = vdupq_n_u8('0');
+ const char *p = dst;
+
+ while (i < chunks_len)
+ {
+ uint8x16x2_t bytes = vld2q_u8((uint8 *) src);
+ uint8x16_t high = bytes.val[0]; /* hex digits for high nibble */
+ uint8x16_t low = bytes.val[1]; /* hex digits for low nibble */
+
+ /* fallback for characters with ASCII values below '0' */
+ uint8x16_t is_below_zero = vorrq_u8(vcltq_u8(high, ascii_zero),
+ vcltq_u8(low, ascii_zero));
+ if (vmaxvq_u8(is_below_zero) != 0)
+ break;
+
+ /* fallback if an invalid hexadecimal digit is found */
+ if (!get_hex_neon(high, &high) || !get_hex_neon(low, &low))
+ break;
+
+ /* form the byte by left-shifting the high nibble and OR-ing it with the low nibble */
+ vst1q_u8((uint8 *) dst, vorrq_u8(vshlq_n_u8(high, 4), low));
+
+ i += 2 * vec_len;
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > i) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* HEX_CODING_AARCH64 */
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index dac372c3bea..8b106d03d33 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -22,6 +22,7 @@ backend_sources += files(
'dbsize.c',
'domains.c',
'encode.c',
+ 'encode_aarch64.c',
'enum.c',
'expandeddatum.c',
'expandedrecord.c',
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d2255..a809fad2771 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,62 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * On AArch64, we can use Neon instructions if the compiler provides access to
+ * them (as indicated by __ARM_NEON). As in simd.h, we assume that all
+ * available 64-bit hardware has Neon support.
+ */
+#if defined(__aarch64__) && defined(__ARM_NEON)
+#define HEX_CODING_AARCH64 1
+#endif
+
+#ifdef HEX_CODING_AARCH64
+extern uint64 hex_encode_optimized(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_optimized(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext);
+#endif
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef HEX_CODING_AARCH64
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef HEX_CODING_AARCH64
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef HEX_CODING_AARCH64
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
[application/octet-stream] v6-0002-SVE-support-for-hex-coding.patch (21.1K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/4-v6-0002-SVE-support-for-hex-coding.patch)
download | inline diff:
From 7330548cc5b5ebdbe5c8fa515f1eb2eebfc7f2c3 Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Thu, 4 Sep 2025 15:33:47 +0530
Subject: [PATCH v6 2/3] SVE support for hex coding
---
config/c-compiler.m4 | 85 +++++++++
configure | 104 +++++++++++
configure.ac | 9 +
meson.build | 81 +++++++++
src/backend/utils/adt/encode_aarch64.c | 231 ++++++++++++++++++++++++-
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 10 +-
7 files changed, 520 insertions(+), 3 deletions(-)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index da40bd6a647..73d12826698 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -798,3 +798,88 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_SVE_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the SVE intrinsic required for hex coding:
+# svsub_x, svcmplt, svsel, svcmpgt, svtbl, svlsr_x, svand_z, svcreate2,
+# svptest_any, svnot_z, svorr_z, svcntb, svld1, svwhilelt_b8, svst2, svld2,
+# svget2, svst1 and svlsl_x.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+
+ char input@<:@64@:>@;
+ char output@<:@128@:>@;
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index 39c68161cec..60354107f87 100755
--- a/configure
+++ b/configure
@@ -17735,6 +17735,110 @@ $as_echo "#define USE_SVE_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_x, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_x, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+
+ char input[64];
+ char output[128];
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_HEX_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index 066e3976c0a..6ca57b8c4a7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2136,6 +2136,15 @@ if test x"$host_cpu" = x"aarch64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_HEX_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index ab8101d67b2..ea5392cfc78 100644
--- a/meson.build
+++ b/meson.build
@@ -2372,6 +2372,87 @@ int main(void)
endif
+###############################################################
+# Check the availability of SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+
+char input[64];
+char output[128];
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+}
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char hextbl[] = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+}
+'''
+
+ if cc.links(prog, name: 'SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_HEX_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
index 7b0412dc255..4f22fb1d6c7 100644
--- a/src/backend/utils/adt/encode_aarch64.c
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -20,8 +20,229 @@
#include <arm_neon.h>
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+
+/*
+ * These are the NEON implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_neon(const char *src, size_t len, char *dst);
+static uint64 hex_decode_neon(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_neon(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * These are the SVE implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * The function pointers are initially set to "choose" functions. These
+ * functions will first set the pointers to the right implementations (based on
+ * what the current CPU supports) and then will call the pointer to fulfill the
+ * caller's request.
+ */
+
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext);
+uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext) = hex_decode_safe_choose;
+
+static inline bool
+check_sve_support(void)
+{
+#ifdef HAVE_ELF_AUX_INFO
+ unsigned long value;
+
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL)
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_neon;
+ hex_decode_optimized = hex_decode_neon;
+ hex_decode_safe_optimized = hex_decode_safe_neon;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint32 vec_len = svcntb();
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svbool_t pred_true = svptrue_b8();
+ size_t chunks_len = len & ~(2 * vec_len - 1); /* process 2 * vec_len bytes per iteration */
+
+ for (size_t i = 0; i < chunks_len; i += 2 * vec_len)
+ {
+ svuint8_t bytes = svld1(pred_true, (uint8 *) src);
+ svuint8_t high = svlsr_x(pred_true, bytes, 4);
+ svuint8_t low = svand_z(pred_true, bytes, 0xF);
+ svuint8x2_t zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred_true, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+
+ /* unrolled */
+ bytes = svld1(pred_true, (uint8 *) src);
+ high = svlsr_x(pred_true, bytes, 4);
+ low = svand_z(pred_true, bytes, 0xF);
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred_true, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ /* process remaining tail bytes */
+ for (size_t i = chunks_len; i < len; i += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64) i, (uint64) len);
+ svuint8_t bytes = svld1(pred, (uint8 *) src);
+ svuint8_t high = svlsr_x(pred, bytes, 4);
+ svuint8_t low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48);
+ svuint8_t upper = svsub_x(pred, vec, 55);
+ svuint8_t lower = svsub_x(pred, vec, 87);
+
+ svbool_t valid_digit = svcmplt(pred, digit, 10);
+ svbool_t valid_upper = svcmplt(pred, upper, 16);
+
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return false;
+
+ *res = svsel(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = svcntb();
+ size_t i = 0;
+ size_t chunks_len = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svbool_t pred = svptrue_b8();
+ const char *p = dst;
+ bool fallback = false;
+
+ while (i < chunks_len)
+ {
+ svuint8x2_t bytes = svld2(pred, (uint8 *) src);
+ svuint8_t high = svget2(bytes, 0);
+ svuint8_t low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ {
+ fallback = true;
+ break;
+ }
+
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ {
+ fallback = true;
+ break;
+ }
+
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ i += 2 * vec_len;
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > i && !fallback) /* can use neon for smaller chunks */
+ return dst - p + hex_decode_safe_neon(src, len - i, dst, escontext);
+
+ if (fallback) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* USE_SVE_HEX_WITH_RUNTIME_CHECK */
+
+/*
+ * If the compiler supports SVE, rename the NEON versions because the
+ * optimized versions are now referenced via function pointers.
+ */
+
uint64
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+static hex_encode_neon(const char *src, size_t len, char *dst)
+#else
hex_encode_optimized(const char *src, size_t len, char *dst)
+#endif
{
const char hextbl[] = "0123456789abcdef";
uint8x16_t hextbl_vec = vld1q_u8((uint8 *) hextbl);
@@ -63,8 +284,6 @@ hex_encode_optimized(const char *src, size_t len, char *dst)
dst += 2 * vec_len;
}
-
-
if (len > chunks_len)
hex_encode_scalar(src, len - chunks_len, dst);
@@ -148,13 +367,21 @@ get_hex_neon(uint8x16_t vec, uint8x16_t *res)
}
uint64
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+static hex_decode_neon(const char *src, size_t len, char *dst)
+#else
hex_decode_optimized(const char *src, size_t len, char *dst)
+#endif
{
return hex_decode_safe_optimized(src, len, dst, NULL);
}
uint64
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+static hex_decode_safe_neon(const char *src, size_t len, char *dst, Node *escontext)
+#else
hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext)
+#endif
{
uint32 vec_len = sizeof(uint8x16_t);
size_t i = 0;
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index c4dc5d72bdb..a6735bdd21f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -678,6 +678,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_HEX_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index a809fad2771..8a80a9ae51f 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -49,7 +49,15 @@ extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
#define HEX_CODING_AARCH64 1
#endif
-#ifdef HEX_CODING_AARCH64
+/*
+ * We can try to use an SVE-optimized hex encode/decode on systems supporting SVE.
+ * For that, we use a function pointer.
+ */
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext);
+#elif HEX_CODING_AARCH64
extern uint64 hex_encode_optimized(const char *src, size_t len, char *dst);
extern uint64 hex_decode_optimized(const char *src, size_t len, char *dst);
extern uint64 hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext);
--
2.34.1
[application/octet-stream] v6-0003-Regression-tests-for-SIMD-hex-coding.patch (7.4K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/5-v6-0003-Regression-tests-for-SIMD-hex-coding.patch)
download | inline diff:
From 2fdd7f1170253984c1b065ac3a0fc43a31997c05 Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Thu, 4 Sep 2025 15:44:19 +0530
Subject: [PATCH v6 3/3] Regression tests for SIMD hex coding
---
src/test/regress/expected/hex_coding.out | 63 ++++++++++++++++++++++++
src/test/regress/parallel_schedule | 5 ++
src/test/regress/sql/hex_coding.sql | 39 +++++++++++++++
3 files changed, 107 insertions(+)
create mode 100644 src/test/regress/expected/hex_coding.out
create mode 100644 src/test/regress/sql/hex_coding.sql
diff --git a/src/test/regress/expected/hex_coding.out b/src/test/regress/expected/hex_coding.out
new file mode 100644
index 00000000000..e6d78fa4876
--- /dev/null
+++ b/src/test/regress/expected/hex_coding.out
@@ -0,0 +1,63 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d');
+ ^
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+ encode
+----------------------------------------------------------------------------------------------------------------------------------
+ ab
+ 01ab
+ deadc0de
+ baadf00d
+ c001c0ffee
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0de
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+(8 rows)
+
+DROP TABLE BYTEA_TABLE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index fbffc67ae60..876a3988ed0 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -109,6 +109,11 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson sqljson_queryfuncs sqljson_jsontable
+# ----------
+# Another group of parallel tests for hex encode/decode
+# ----------
+test: hex_coding
+
# ----------
# Another group of parallel tests
# with depends on create_misc
diff --git a/src/test/regress/sql/hex_coding.sql b/src/test/regress/sql/hex_coding.sql
new file mode 100644
index 00000000000..97c51b62e90
--- /dev/null
+++ b/src/test/regress/sql/hex_coding.sql
@@ -0,0 +1,39 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+
+DROP TABLE BYTEA_TABLE;
--
2.34.1
[image/svg+xml] bytea_read_hex_encode_sve.svg (292.8K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/6-bytea_read_hex_encode_sve.svg)
download | view image
[image/svg+xml] bytea_read_hex_encode_v17.svg (287.7K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/7-bytea_read_hex_encode_v17.svg)
download | view image
[image/svg+xml] bytea_read_hex_encode_v18.svg (255.1K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/8-bytea_read_hex_encode_v18.svg)
download | view image
[image/svg+xml] bytea_write_hex_decode_sve.svg (325.4K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/9-bytea_write_hex_decode_sve.svg)
download | view image
[image/svg+xml] bytea_write_hex_decode_v18.svg (280.4K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/10-bytea_write_hex_decode_v18.svg)
download | view image
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-11 03:12 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-11 03:12 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: John Naylor <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 04, 2025 at 02:55:50PM +0000, [email protected] wrote:
>> I see that there was some discussion about a Neon implementation upthread,
>> but I'm not sure we concluded anything. For popcount, we first added a
>> Neon version before adding the SVE version, which required more complicated
>> configure/runtime checks. Presumably Neon is available on more hardware
>> than SVE, so that could be a good place to start here, too.
>
> We have added the Neon versions of hex encode/decode.
Thanks. I noticed that this stuff is simple enough that we can use
port/simd.h (with a few added functions). This is especially nice because
it takes care of x86, too. The performance gains look similar to what you
reported for v6:
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 13 | 6 | 54
64 | 34 | 9 | 74
256 | 93 | 25 | 73
1024 | 281 | 78 | 72
4096 | 1086 | 227 | 79
16384 | 4382 | 927 | 79
65536 | 17455 | 3608 | 79
x86
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 10 | 7 | 30
64 | 29 | 9 | 69
256 | 81 | 21 | 74
1024 | 286 | 66 | 77
4096 | 1106 | 253 | 77
16384 | 4383 | 980 | 78
65536 | 17491 | 3886 | 78
I've only modified hex_encode() for now, but I'm optimistic that we can do
something similar for hex_decode().
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-11 10:43 [email protected] <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-09-11 10:43 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: John Naylor <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> Thanks. I noticed that this stuff is simple enough that we can use
> port/simd.h (with a few added functions). This is especially nice because
> it takes care of x86, too. The performance gains look similar to what you
> reported for v6:
This looks good, much cleaner.
One possible improvement would be to use a vectorized table lookup instead of compare and add. I compared v6 and v7 Neon versions, and v6 is always faster.
I’m not sure if SSE2 has a table lookup similar to Neon.
arm - m7g.4xlarge
buf | v6-Neon| v7-Neon| % diff
-------+--------+--------+--------
64 | 6.16 | 8.57 | 28.07
128 | 11.37 | 15.77 | 27.87
256 | 18.54 | 30.28 | 38.77
512 | 33.98 | 62.15 | 45.33
1024 | 64.46 | 117.55 | 45.16
2048 | 124.28 | 254.86 | 51.24
4096 | 243.47 | 509.23 | 52.19
8192 | 487.34 | 953.81 | 48.91
-----
Chiranmoy
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-11 15:24 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-11 15:24 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: John Naylor <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 11, 2025 at 10:43:56AM +0000, [email protected] wrote:
> One possible improvement would be to use a vectorized table lookup
> instead of compare and add. I compared v6 and v7 Neon versions, and v6 is
> always faster. I’m not sure if SSE2 has a table lookup similar to Neon.
I'm not finding a simple way to do that kind of table lookup in SSE2. Part
of the reason v6 performs better is because you've unrolled the loop to
process 2 vector's worth of input data in each iteration. This trades
performance with smaller inputs for gains with larger ones. But even if I
do something similar for v7, v6 still wins most of the time.
My current philosophy with this stuff is to favor simplicity,
maintainability, portability, etc. over extracting the absolute maximum
amount of performance gain, so I think we should proceed with the simd.h
approach. But I'm curious how others feel about this.
v8 is an attempt to fix the casting error on MSVC.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-11 15:32 Tom Lane <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Tom Lane @ 2025-09-11 15:32 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Nathan Bossart <[email protected]> writes:
> My current philosophy with this stuff is to favor simplicity,
> maintainability, portability, etc. over extracting the absolute maximum
> amount of performance gain, so I think we should proceed with the simd.h
> approach. But I'm curious how others feel about this.
+1. The maintainability aspect is critical over the long run.
Also, there's a very real danger of optimizing for the specific
hardware and test case you are working with, leading to actually
worse performance with future hardware.
regards, tom lane
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-12 18:49 [email protected] <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: [email protected] @ 2025-09-12 18:49 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; +Cc: John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> My current philosophy with this stuff is to favor simplicity,
> maintainability, portability, etc. over extracting the absolute maximum
> amount of performance gain, so I think we should proceed with the simd.h
> approach. But I'm curious how others feel about this.
> +1. The maintainability aspect is critical over the long run.
> Also, there's a very real danger of optimizing for the specific
> hardware and test case you are working with, leading to actually
> worse performance with future hardware.
Using simd.h does make it easier to maintain.
Is there a plan to upgrade simd.h to use SSE4 or SSSE3 in the future?
Since SSE2 is much older, it lacks some of the more specialized intrinsics.
For example, vectorized table lookup can be implemented via [0], and
it’s available in SSSE3 and later x86 instruction sets.
[0] https://www.felixcloutier.com/x86/pshufb
-----
Chiranmoy
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-12 21:30 Nathan Bossart <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-12 21:30 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Sep 12, 2025 at 06:49:01PM +0000, [email protected] wrote:
> Using simd.h does make it easier to maintain. Is there a plan to upgrade
> simd.h to use SSE4 or SSSE3 in the future? Since SSE2 is much older, it
> lacks some of the more specialized intrinsics. For example, vectorized
> table lookup can be implemented via [0], and it’s available in SSSE3 and
> later x86 instruction sets.
There have been a couple of discussions about the possibility of requiring
x86-64-v2 for Postgres, but I'm not aware of any serious efforts in that
area.
I've attached a new version of the patch with a simd.h version of
hex_decode(). Here are the numbers:
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 22 | 23 | -5
64 | 61 | 23 | 62
256 | 158 | 47 | 70
1024 | 542 | 122 | 77
4096 | 2103 | 429 | 80
16384 | 8548 | 1673 | 80
65536 | 34663 | 6738 | 81
x86
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 13 | 14 | -8
64 | 42 | 15 | 64
256 | 126 | 42 | 67
1024 | 461 | 149 | 68
4096 | 1802 | 576 | 68
16384 | 7166 | 2280 | 68
65536 | 28625 | 9108 | 68
A couple of notes:
* For hex_decode(), we just give up on the SIMD path and fall back on the
scalar path as soon as we see anything outside [0-9A-Za-z]. I suspect
this might introduce a regression for inputs of ~32 to ~64 bytes that
include whitespace (which must be skipped) or invalid characters, but I
don't whether those inputs are common or whether we care.
* The code makes some assumptions about endianness that might not be true
everywhere, but I've yet to dig into this further.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-22 20:05 Nathan Bossart <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-22 20:05 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Sep 12, 2025 at 04:30:21PM -0500, Nathan Bossart wrote:
> I've attached a new version of the patch with a simd.h version of
> hex_decode(). Here are the numbers:
I was able to improve the hex_decode() implementation a bit.
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 11 | 11 | 0
64 | 38 | 7 | 82
256 | 133 | 18 | 86
1024 | 513 | 67 | 87
4096 | 2037 | 271 | 87
16384 | 8326 | 1103 | 87
65536 | 34550 | 4475 | 87
x86
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 8 | 9 | -13
64 | 38 | 7 | 82
256 | 121 | 24 | 80
1024 | 457 | 91 | 80
4096 | 1797 | 356 | 80
16384 | 7161 | 1411 | 80
65536 | 28620 | 5632 | 80
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-23 19:02 Nathan Bossart <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-23 19:02 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Sep 22, 2025 at 03:05:44PM -0500, Nathan Bossart wrote:
> I was able to improve the hex_decode() implementation a bit.
I took a closer look at how hex_decode() performs with smaller inputs.
There are some small regressions, so I tried fixing them by adding the
following to the beginning of the function:
if (likely(tail_idx == 0))
return hex_decode_safe_scalar(src, len, dst, escontext);
This helped a little, but it mostly just slowed things down for larger
inputs on AArch64:
arm
buf | HEAD | patch | fix
-------+-------+-------+-------
2 | 4 | 6 | 4
4 | 6 | 7 | 7
8 | 8 | 8 | 8
16 | 11 | 12 | 11
32 | 18 | 5 | 6
64 | 38 | 7 | 8
256 | 134 | 18 | 24
1024 | 514 | 67 | 100
4096 | 2072 | 280 | 389
16384 | 8409 | 1126 | 1537
65536 | 34704 | 4498 | 6128
x86
buf | HEAD | patch | fix
-------+-------+-------+-------
2 | 2 | 2 | 2
4 | 3 | 3 | 3
8 | 4 | 4 | 4
16 | 8 | 9 | 8
32 | 23 | 5 | 5
64 | 37 | 7 | 7
256 | 122 | 24 | 24
1024 | 457 | 91 | 92
4096 | 1798 | 357 | 358
16384 | 7161 | 1411 | 1416
65536 | 28621 | 5630 | 5653
I didn't do this test for hex_encode(), but I'd expect it to follow a
similar pattern. I'm tempted to suggest that these regressions are within
tolerable levels and to forge on with v10. In any case, IMHO this patch is
approaching committable quality, so I'd be grateful for any feedback.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-24 03:59 John Naylor <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: John Naylor @ 2025-09-24 03:59 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 24, 2025 at 2:02 AM Nathan Bossart <[email protected]> wrote:
>
> On Mon, Sep 22, 2025 at 03:05:44PM -0500, Nathan Bossart wrote:
> > I was able to improve the hex_decode() implementation a bit.
>
> I took a closer look at how hex_decode() performs with smaller inputs.
> There are some small regressions, so I tried fixing them by adding the
> following to the beginning of the function:
>
> if (likely(tail_idx == 0))
> return hex_decode_safe_scalar(src, len, dst, escontext);
>
> This helped a little, but it mostly just slowed things down for larger
> inputs on AArch64:
> I didn't do this test for hex_encode(), but I'd expect it to follow a
> similar pattern. I'm tempted to suggest that these regressions are within
> tolerable levels and to forge on with v10.
My first thought is, I'd hazard a guess that short byteas are much
less common than short strings.
My second thought is, the decode case is not that critical. From the
end-to-end tests above, the speed of the decode case had a relatively
small global effect compared to the encode case (Perhaps because reads
are cheaper than writes).
+ if (unlikely(!hex_decode_simd_helper(srcv, &dstv1)))
+ break;
But if you really want to do something here, sprinkling "(un)likely"'s
here seems like solving the wrong problem (even if they make any
difference), since the early return is optimizing for exceptional
conditions. In other places (cf. the UTF8 string verifier), we
accumulate errors, and only if we have them at the end do we restart
from the beginning with the slow error-checking path that can show the
user the offending input.
> In any case, IMHO this patch is
> approaching committable quality, so I'd be grateful for any feedback.
+vector8_sssub(const Vector8 v1, const Vector8 v2)
It's hard to parse "sss", so maybe we can borrow an Intel-ism and use
"iss" for the signed case?
+/* vector manipulation */
+#ifndef USE_NO_SIMD
+static inline Vector8 vector8_interleave_low(const Vector8 v1, const
Vector8 v2);
+static inline Vector8 vector8_interleave_high(const Vector8 v1, const
Vector8 v2);
+static inline Vector8 vector8_pack_16(const Vector8 v1, const Vector8 v2);
+static inline Vector32 vector32_shift_left_nibble(const Vector32 v1);
+static inline Vector32 vector32_shift_right_nibble(const Vector32 v1);
+static inline Vector32 vector32_shift_right_byte(const Vector32 v1);
Do we need declarations for these? I recall that the existing
declarations are there for functions that are also used internally.
The nibble/byte things are rather specific. Wouldn't it be more
logical to expose the already-generic shift operations and let the
caller say by how much? Or does the compiler refuse because the
intrinsic doesn't get an immediate value? Some are like that, but I'm
not sure about these. If so, that's annoying and I wonder if there's a
workaround.
+vector8_has_ge(const Vector8 v, const uint8 c)
+{
+#ifdef USE_SSE2
+ Vector8 umax = _mm_max_epu8(v, vector8_broadcast(c));
+ Vector8 cmpe = _mm_cmpeq_epi8(umax, v);
+
+ return vector8_is_highbit_set(cmpe);
We take pains to avoid signed comparison on unsigned input for the
"le" case, and I don't see why it's okay here.
Do the regression tests have long enough cases that test exceptional
paths, like invalid bytes and embedded whitespace? If not, we need
some.
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-24 21:40 Nathan Bossart <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-24 21:40 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 24, 2025 at 10:59:38AM +0700, John Naylor wrote:
> + if (unlikely(!hex_decode_simd_helper(srcv, &dstv1)))
> + break;
>
> But if you really want to do something here, sprinkling "(un)likely"'s
> here seems like solving the wrong problem (even if they make any
> difference), since the early return is optimizing for exceptional
> conditions. In other places (cf. the UTF8 string verifier), we
> accumulate errors, and only if we have them at the end do we restart
> from the beginning with the slow error-checking path that can show the
> user the offending input.
I switched to an accumulator approach in v11.
> +vector8_sssub(const Vector8 v1, const Vector8 v2)
>
> It's hard to parse "sss", so maybe we can borrow an Intel-ism and use
> "iss" for the signed case?
Done.
> +/* vector manipulation */
> +#ifndef USE_NO_SIMD
> +static inline Vector8 vector8_interleave_low(const Vector8 v1, const
> Vector8 v2);
> +static inline Vector8 vector8_interleave_high(const Vector8 v1, const
> Vector8 v2);
> +static inline Vector8 vector8_pack_16(const Vector8 v1, const Vector8 v2);
> +static inline Vector32 vector32_shift_left_nibble(const Vector32 v1);
> +static inline Vector32 vector32_shift_right_nibble(const Vector32 v1);
> +static inline Vector32 vector32_shift_right_byte(const Vector32 v1);
>
> Do we need declarations for these? I recall that the existing
> declarations are there for functions that are also used internally.
Removed.
> The nibble/byte things are rather specific. Wouldn't it be more
> logical to expose the already-generic shift operations and let the
> caller say by how much? Or does the compiler refuse because the
> intrinsic doesn't get an immediate value? Some are like that, but I'm
> not sure about these. If so, that's annoying and I wonder if there's a
> workaround.
Yeah, the compiler refuses unless the value is an integer literal. I
thought of using a switch statement to cover all the values used in-tree,
but I didn't like that, either.
> +vector8_has_ge(const Vector8 v, const uint8 c)
> +{
> +#ifdef USE_SSE2
> + Vector8 umax = _mm_max_epu8(v, vector8_broadcast(c));
> + Vector8 cmpe = _mm_cmpeq_epi8(umax, v);
> +
> + return vector8_is_highbit_set(cmpe);
>
> We take pains to avoid signed comparison on unsigned input for the
> "le" case, and I don't see why it's okay here.
_mm_max_epu8() does unsigned comparisons, I think...
> Do the regression tests have long enough cases that test exceptional
> paths, like invalid bytes and embedded whitespace? If not, we need
> some.
Added.
I've also fixed builds on gcc/arm64, as discussed elsewhere [0]. Here are
the current numbers on my laptop:
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
2 | 4 | 4 | 0
4 | 6 | 6 | 0
8 | 8 | 8 | 0
16 | 11 | 12 | -9
32 | 18 | 5 | 72
64 | 38 | 6 | 84
256 | 134 | 17 | 87
1024 | 513 | 63 | 88
4096 | 2081 | 262 | 87
16384 | 8524 | 1058 | 88
65536 | 34731 | 4224 | 88
[0] https://postgr.es/m/aNQtN89VW8z-yo3B%40nathan
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-25 14:16 John Naylor <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: John Naylor @ 2025-09-25 14:16 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 25, 2025 at 4:40 AM Nathan Bossart <[email protected]> wrote:
>
> On Wed, Sep 24, 2025 at 10:59:38AM +0700, John Naylor wrote:
> > + if (unlikely(!hex_decode_simd_helper(srcv, &dstv1)))
> > + break;
> >
> > But if you really want to do something here, sprinkling "(un)likely"'s
> > here seems like solving the wrong problem (even if they make any
> > difference), since the early return is optimizing for exceptional
> > conditions. In other places (cf. the UTF8 string verifier), we
> > accumulate errors, and only if we have them at the end do we restart
> > from the beginning with the slow error-checking path that can show the
> > user the offending input.
>
> I switched to an accumulator approach in v11.
Looks good to me.
+ if (unlikely(!success))
+ i = 0;
This is after the main loop exits, and the cold path is literally one
instruction, so the motivation is not apparent to me.
> > The nibble/byte things are rather specific. Wouldn't it be more
> > logical to expose the already-generic shift operations and let the
> > caller say by how much? Or does the compiler refuse because the
> > intrinsic doesn't get an immediate value? Some are like that, but I'm
> > not sure about these. If so, that's annoying and I wonder if there's a
> > workaround.
>
> Yeah, the compiler refuses unless the value is an integer literal. I
> thought of using a switch statement to cover all the values used in-tree,
> but I didn't like that, either.
Neither option is great, but I mildly lean towards keeping it internal
with "switch" or whatever: By putting the burden of specifying shift
amounts on separately named functions we run a risk of combinatorial
explosion in function names.
If you feel otherwise, I'd at least use actual numbers:
"shift_left_nibble" is an awkward way to say "shift left by 4 bits"
anyway, and also after "byte" and "nibble" there are not many good
English words to convey the operand amount. It's very possible that
needing other shift amounts will never come up, though.
> > +vector8_has_ge(const Vector8 v, const uint8 c)
> > +{
> > +#ifdef USE_SSE2
> > + Vector8 umax = _mm_max_epu8(v, vector8_broadcast(c));
> > + Vector8 cmpe = _mm_cmpeq_epi8(umax, v);
> > +
> > + return vector8_is_highbit_set(cmpe);
> >
> > We take pains to avoid signed comparison on unsigned input for the
> > "le" case, and I don't see why it's okay here.
>
> _mm_max_epu8() does unsigned comparisons, I think...
Ah, I confused myself about what the LE case was avoiding, namely
signed LE, not signed equality on something else.
(Separately, now I'm wondering if we can do the same for
vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
would allow getting rid of )
> > Do the regression tests have long enough cases that test exceptional
> > paths, like invalid bytes and embedded whitespace? If not, we need
> > some.
>
> Added.
Seems comprehensive enough at a glance.
Other comments:
+ * back together to form the final hex-encoded string. It might be
+ * possible to squeeze out a little more gain by manually unrolling the
+ * loop, but for now we don't bother.
My position (and I think the community agrees) is that manual
unrolling is a rare desperation move that has to be justified, so we
don't need to mention its lack.
+ * Some compilers are picky about casts to the same underlying type, and others
+ * are picky about implicit conversions with vector types. This function does
+ * the same thing as vector32_broadcast(), but it returns a Vector8 and is
+ * carefully crafted to avoid compiler indigestion.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_broadcast_u32(const uint32 c)
+{
+#ifdef USE_SSE2
+ return vector32_broadcast(c);
+#elif defined(USE_NEON)
+ return (Vector8) vector32_broadcast(c);
+#endif
+}
I'm ambivalent about this: The use case doesn't seem well motivated,
since I don't know why we'd actually need to both broadcast arbitrary
integers and also view the result as bytes. Setting arbitrary bytes is
what we're really doing, and would be more likely be useful in the
future (attached, only tested on x86, and I think part of the
strangeness is the endianness you mentioned above). On the other hand,
the Arm workaround results in awful generated code compared to what
you have here. Since the "set" should be hoisted out of the outer
loop, and we already rely on this pattern for vector8_highbit_mask
anyway, it might be tolerable, and we can reduce the pain with bitwise
NOT.
+/*
+ * Pack 16-bit elements in the given vectors into a single vector of 8-bit
+ * elements. NB: The upper 8-bits of each 16-bit element must be zeros, else
+ * this will produce different results on different architectures.
+ */
v10 asserted this requirement -- that still seems like a good thing?
--
John Naylor
Amazon Web Services
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 7ba92c2c481..eb932d4bd8a 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -317,6 +317,9 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ // TODO: set one and use bitwise NOT for the other
+ Vector8 maskupper = vector8_set(0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0);
+ Vector8 masklower = vector8_set(0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff);
Vector8 msk;
bool ret;
@@ -334,9 +337,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0xff00ff00));
+ msk = vector8_and(*dst, masklower);
msk = vector8_shift_right_byte(msk);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x00ff00ff));
+ *dst = vector8_and(*dst, maskupper);
*dst = vector8_shift_left_nibble(*dst);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 531d8b8b6d1..56e810ce081 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,33 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/*
+ * Populate a vector element-wise with the arguments.
+ */
+#ifndef USE_NO_SIMD
+#if defined(USE_NEON)
+// from a patch by Thomas Munro
+static inline Vector8
+vector8_set(uint8 v0, uint8 v1, uint8 v2, uint8 v3,
+ uint8 v4, uint8 v5, uint8 v6, uint8 v7,
+ uint8 v8, uint8 v9, uint8 v10, uint8 v11,
+ uint8 v12, uint8 v13, uint8 v14, uint8 v15)
+{
+ uint8 pg_attribute_aligned(16) values[16] = {
+ v0, v1, v2, v3,
+ v4, v5, v6, v7,
+ v8, v9, v10, v11,
+ v12, v13, v14, v15
+ };
+ return vld1q_u8(values);
+}
+#elif defined(USE_SSE2)
+#ifndef vector8_set
+#define vector8_set(...) _mm_setr_epi8(__VA_ARGS__)
+#endif
+#endif
+#endif /* ! USE_NO_SIMD */
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -368,6 +395,7 @@ vector8_highbit_mask(const Vector8 v)
* returns a uint64, making it inconvenient to combine mask values from
* multiple vectors.
*/
+ // TODO: use vector8_set
static const uint8 mask[16] = {
1 << 0, 1 << 1, 1 << 2, 1 << 3,
1 << 4, 1 << 5, 1 << 6, 1 << 7,
Attachments:
[text/plain] vector8_set.patch.txt (2.4K, ../../CANWCAZb9FhrWf_2OzwXTK_sQeXm0nPz2joimukB-7pVF0SXxXw@mail.gmail.com/2-vector8_set.patch.txt)
download | inline diff:
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 7ba92c2c481..eb932d4bd8a 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -317,6 +317,9 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ // TODO: set one and use bitwise NOT for the other
+ Vector8 maskupper = vector8_set(0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0);
+ Vector8 masklower = vector8_set(0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff);
Vector8 msk;
bool ret;
@@ -334,9 +337,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0xff00ff00));
+ msk = vector8_and(*dst, masklower);
msk = vector8_shift_right_byte(msk);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x00ff00ff));
+ *dst = vector8_and(*dst, maskupper);
*dst = vector8_shift_left_nibble(*dst);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 531d8b8b6d1..56e810ce081 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,33 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/*
+ * Populate a vector element-wise with the arguments.
+ */
+#ifndef USE_NO_SIMD
+#if defined(USE_NEON)
+// from a patch by Thomas Munro
+static inline Vector8
+vector8_set(uint8 v0, uint8 v1, uint8 v2, uint8 v3,
+ uint8 v4, uint8 v5, uint8 v6, uint8 v7,
+ uint8 v8, uint8 v9, uint8 v10, uint8 v11,
+ uint8 v12, uint8 v13, uint8 v14, uint8 v15)
+{
+ uint8 pg_attribute_aligned(16) values[16] = {
+ v0, v1, v2, v3,
+ v4, v5, v6, v7,
+ v8, v9, v10, v11,
+ v12, v13, v14, v15
+ };
+ return vld1q_u8(values);
+}
+#elif defined(USE_SSE2)
+#ifndef vector8_set
+#define vector8_set(...) _mm_setr_epi8(__VA_ARGS__)
+#endif
+#endif
+#endif /* ! USE_NO_SIMD */
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -368,6 +395,7 @@ vector8_highbit_mask(const Vector8 v)
* returns a uint64, making it inconvenient to combine mask values from
* multiple vectors.
*/
+ // TODO: use vector8_set
static const uint8 mask[16] = {
1 << 0, 1 << 1, 1 << 2, 1 << 3,
1 << 4, 1 << 5, 1 << 6, 1 << 7,
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-25 18:50 Nathan Bossart <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-09-25 18:50 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 25, 2025 at 09:16:35PM +0700, John Naylor wrote:
> + if (unlikely(!success))
> + i = 0;
>
> This is after the main loop exits, and the cold path is literally one
> instruction, so the motivation is not apparent to me.
Removed. I was thinking about smaller inputs when I added this, but it
probably makes little difference.
>> Yeah, the compiler refuses unless the value is an integer literal. I
>> thought of using a switch statement to cover all the values used in-tree,
>> but I didn't like that, either.
>
> Neither option is great, but I mildly lean towards keeping it internal
> with "switch" or whatever: By putting the burden of specifying shift
> amounts on separately named functions we run a risk of combinatorial
> explosion in function names.
Done.
> (Separately, now I'm wondering if we can do the same for
> vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
> would allow getting rid of )
I think so. I doubt there's any performance advantage, but it could be
nice for code cleanup. (I'm assuming you meant to say vector8_ssub
(renamed to vector8_ussub() in the patch) after "getting rid of.") I'll
do this in the related patch in the "couple of small patches for simd.h"
thread.
> + * back together to form the final hex-encoded string. It might be
> + * possible to squeeze out a little more gain by manually unrolling the
> + * loop, but for now we don't bother.
>
> My position (and I think the community agrees) is that manual
> unrolling is a rare desperation move that has to be justified, so we
> don't need to mention its lack.
Removed.
> + * Some compilers are picky about casts to the same underlying type, and others
> + * are picky about implicit conversions with vector types. This function does
> + * the same thing as vector32_broadcast(), but it returns a Vector8 and is
> + * carefully crafted to avoid compiler indigestion.
> + */
> +#ifndef USE_NO_SIMD
> +static inline Vector8
> +vector8_broadcast_u32(const uint32 c)
> +{
> +#ifdef USE_SSE2
> + return vector32_broadcast(c);
> +#elif defined(USE_NEON)
> + return (Vector8) vector32_broadcast(c);
> +#endif
> +}
>
> I'm ambivalent about this: The use case doesn't seem well motivated,
> since I don't know why we'd actually need to both broadcast arbitrary
> integers and also view the result as bytes. Setting arbitrary bytes is
> what we're really doing, and would be more likely be useful in the
> future (attached, only tested on x86, and I think part of the
> strangeness is the endianness you mentioned above). On the other hand,
> the Arm workaround results in awful generated code compared to what
> you have here. Since the "set" should be hoisted out of the outer
> loop, and we already rely on this pattern for vector8_highbit_mask
> anyway, it might be tolerable, and we can reduce the pain with bitwise
> NOT.
I think I disagree on this one. We're not broadcasting arbitrary bytes for
every vector element, we're broadcasting a patten of bytes that happens to
be wider than the element size. I would expect this to be a relatively
common use-case. Furthermore, the "set" API is closely tethered to the
vector size, which is fine for SSE2/Neon but may not work down the road
(not to mention the USE_NO_SIMD path). Also, the bitwise NOT approach
won't work because we need to use 0x0f000f00 and 0x000f000f to avoid
angering the assertion in vector8_pack_16(), as mentioned below.
> +/*
> + * Pack 16-bit elements in the given vectors into a single vector of 8-bit
> + * elements. NB: The upper 8-bits of each 16-bit element must be zeros, else
> + * this will produce different results on different architectures.
> + */
>
> v10 asserted this requirement -- that still seems like a good thing?
I had removed that because I worried the accumulator approach would cause
it to fail (it does), but looking again, that's easy enough to work around.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-09-29 08:45 John Naylor <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: John Naylor @ 2025-09-29 08:45 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Sep 26, 2025 at 1:50 AM Nathan Bossart <[email protected]> wrote:
>
> On Thu, Sep 25, 2025 at 09:16:35PM +0700, John Naylor wrote:
> > (Separately, now I'm wondering if we can do the same for
> > vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
> > would allow getting rid of )
>
> I think so. I doubt there's any performance advantage, but it could be
> nice for code cleanup. (I'm assuming you meant to say vector8_ssub
> (renamed to vector8_ussub() in the patch) after "getting rid of.")
Yes right, sorry. And it seems good to do such cleanup first, since it
doesn't make sense to rename something that is about to be deleted.
> I think I disagree on this one. We're not broadcasting arbitrary bytes for
> every vector element, we're broadcasting a patten of bytes that happens to
> be wider than the element size. I would expect this to be a relatively
> common use-case.
That's probably true. I'm still worried that the hack for working
around compiler pickiness (while nice enough in it's current form)
might break at some point and require awareness of compiler versions.
Hmm, for this case, we can sidestep the maintainability questions
entirely by instead using the new interleave functions to build the
masks:
vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f))
vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero())
This generates identical code as v12 on Arm and is not bad on x86.
What do you think of the attached?
While looking around again, it looks like the "msk" variable isn't a
mask like the implies to me. Not sure of a better name because I'm not
sure what it represents aside from a temp variable.
+#elif defined(USE_NEON)
+ switch (i)
+ {
+ case 4:
+ return (Vector8) vshrq_n_u32((Vector32) v1, 4);
+ case 8:
+ return (Vector8) vshrq_n_u32((Vector32) v1, 8);
+ default:
+ pg_unreachable();
+ return vector8_broadcast(0);
+ }
This is just a compiler hint, so if the input is not handled I think
it will return the wrong answer rather than alerting the developer, so
we probabaly want "Assert(false)" here.
Other than that, the pack/unpack functions could use some
documentation about which parameter is low/high.
--
John Naylor
Amazon Web Services
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 87126a003b3..db734481a60 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -315,6 +315,8 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ Vector8 mask_hi = vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f));
+ Vector8 mask_lo = vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero());
Vector8 msk;
bool ret;
@@ -332,9 +334,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0x0f000f00));
+ msk = vector8_and(*dst, mask_hi);
msk = vector8_shift_right(msk, 8);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x000f000f));
+ *dst = vector8_and(*dst, mask_lo);
*dst = vector8_shift_left(*dst, 4);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 0261179e9e7..e316233b7aa 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,17 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/* return a zeroed register */
+static inline Vector8
+vector8_zero()
+{
+#if defined(USE_SSE2)
+ return _mm_setzero_si128();
+#elif defined(USE_NEON)
+ return vmovq_n_u8(0);
+#endif
+}
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -170,24 +181,6 @@ vector32_broadcast(const uint32 c)
}
#endif /* ! USE_NO_SIMD */
-/*
- * Some compilers are picky about casts to the same underlying type, and others
- * are picky about implicit conversions with vector types. This function does
- * the same thing as vector32_broadcast(), but it returns a Vector8 and is
- * carefully crafted to avoid compiler indigestion.
- */
-#ifndef USE_NO_SIMD
-static inline Vector8
-vector8_broadcast_u32(const uint32 c)
-{
-#ifdef USE_SSE2
- return vector32_broadcast(c);
-#elif defined(USE_NEON)
- return (Vector8) vector32_broadcast(c);
-#endif
-}
-#endif /* ! USE_NO_SIMD */
-
/*
* Return true if any elements in the vector are equal to the given scalar.
*/
@@ -577,8 +570,9 @@ vector8_interleave_high(const Vector8 v1, const Vector8 v2)
static inline Vector8
vector8_pack_16(const Vector8 v1, const Vector8 v2)
{
- Vector32 mask PG_USED_FOR_ASSERTS_ONLY = vector32_broadcast(0xff00ff00);
+ Vector8 mask PG_USED_FOR_ASSERTS_ONLY;
+ mask = vector8_interleave_low(vector8_zero(), vector8_broadcast(0xff));
Assert(!vector8_has_ge(vector8_and(v1, mask), 1));
Assert(!vector8_has_ge(vector8_and(v2, mask), 1));
#ifdef USE_SSE2
Attachments:
[text/plain] interleave-mask.patch.txt (2.7K, ../../CANWCAZZMaxfO3CLr1_ViUZMknp_M9iJ0LJbw63E4xMRanGiSqg@mail.gmail.com/2-interleave-mask.patch.txt)
download | inline diff:
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 87126a003b3..db734481a60 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -315,6 +315,8 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ Vector8 mask_hi = vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f));
+ Vector8 mask_lo = vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero());
Vector8 msk;
bool ret;
@@ -332,9 +334,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0x0f000f00));
+ msk = vector8_and(*dst, mask_hi);
msk = vector8_shift_right(msk, 8);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x000f000f));
+ *dst = vector8_and(*dst, mask_lo);
*dst = vector8_shift_left(*dst, 4);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 0261179e9e7..e316233b7aa 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,17 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/* return a zeroed register */
+static inline Vector8
+vector8_zero()
+{
+#if defined(USE_SSE2)
+ return _mm_setzero_si128();
+#elif defined(USE_NEON)
+ return vmovq_n_u8(0);
+#endif
+}
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -170,24 +181,6 @@ vector32_broadcast(const uint32 c)
}
#endif /* ! USE_NO_SIMD */
-/*
- * Some compilers are picky about casts to the same underlying type, and others
- * are picky about implicit conversions with vector types. This function does
- * the same thing as vector32_broadcast(), but it returns a Vector8 and is
- * carefully crafted to avoid compiler indigestion.
- */
-#ifndef USE_NO_SIMD
-static inline Vector8
-vector8_broadcast_u32(const uint32 c)
-{
-#ifdef USE_SSE2
- return vector32_broadcast(c);
-#elif defined(USE_NEON)
- return (Vector8) vector32_broadcast(c);
-#endif
-}
-#endif /* ! USE_NO_SIMD */
-
/*
* Return true if any elements in the vector are equal to the given scalar.
*/
@@ -577,8 +570,9 @@ vector8_interleave_high(const Vector8 v1, const Vector8 v2)
static inline Vector8
vector8_pack_16(const Vector8 v1, const Vector8 v2)
{
- Vector32 mask PG_USED_FOR_ASSERTS_ONLY = vector32_broadcast(0xff00ff00);
+ Vector8 mask PG_USED_FOR_ASSERTS_ONLY;
+ mask = vector8_interleave_low(vector8_zero(), vector8_broadcast(0xff));
Assert(!vector8_has_ge(vector8_and(v1, mask), 1));
Assert(!vector8_has_ge(vector8_and(v2, mask), 1));
#ifdef USE_SSE2
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-10-02 17:33 Nathan Bossart <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-10-02 17:33 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Sep 29, 2025 at 03:45:27PM +0700, John Naylor wrote:
> On Fri, Sep 26, 2025 at 1:50 AM Nathan Bossart <[email protected]> wrote:
>> On Thu, Sep 25, 2025 at 09:16:35PM +0700, John Naylor wrote:
>> > (Separately, now I'm wondering if we can do the same for
>> > vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
>> > would allow getting rid of )
>>
>> I think so. I doubt there's any performance advantage, but it could be
>> nice for code cleanup. (I'm assuming you meant to say vector8_ssub
>> (renamed to vector8_ussub() in the patch) after "getting rid of.")
>
> Yes right, sorry. And it seems good to do such cleanup first, since it
> doesn't make sense to rename something that is about to be deleted.
Will do. I'll plan on committing the other patch [0] soon.
> Hmm, for this case, we can sidestep the maintainability questions
> entirely by instead using the new interleave functions to build the
> masks:
>
> vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f))
> vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero())
>
> This generates identical code as v12 on Arm and is not bad on x86.
> What do you think of the attached?
WFM
> While looking around again, it looks like the "msk" variable isn't a
> mask like the implies to me. Not sure of a better name because I'm not
> sure what it represents aside from a temp variable.
Renamed to "tmp".
> +#elif defined(USE_NEON)
> + switch (i)
> + {
> + case 4:
> + return (Vector8) vshrq_n_u32((Vector32) v1, 4);
> + case 8:
> + return (Vector8) vshrq_n_u32((Vector32) v1, 8);
> + default:
> + pg_unreachable();
> + return vector8_broadcast(0);
> + }
>
> This is just a compiler hint, so if the input is not handled I think
> it will return the wrong answer rather than alerting the developer, so
> we probabaly want "Assert(false)" here.
Fixed.
> Other than that, the pack/unpack functions could use some
> documentation about which parameter is low/high.
Added.
[0] https://postgr.es/m/attachment/182185/v3-0001-Optimize-vector8_has_le-on-AArch64.patch
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-10-03 07:36 John Naylor <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: John Naylor @ 2025-10-03 07:36 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Oct 3, 2025 at 12:33 AM Nathan Bossart <[email protected]> wrote:
> [v13]
LGTM, but I went back and checked if vector8_zero() actually does
anything different than vector8_boadcast(0), and in fact it doesn't on
compilers we support for either x86 or Arm. I pulled the former out
from older work, but it seems irrelevant now. Pardon the noise.
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-10-03 20:33 Nathan Bossart <[email protected]>
parent: John Naylor <[email protected]>
0 siblings, 1 reply; 36+ messages in thread
From: Nathan Bossart @ 2025-10-03 20:33 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Oct 03, 2025 at 02:36:47PM +0700, John Naylor wrote:
> LGTM, but I went back and checked if vector8_zero() actually does
> anything different than vector8_boadcast(0), and in fact it doesn't on
> compilers we support for either x86 or Arm. I pulled the former out
> from older work, but it seems irrelevant now. Pardon the noise.
Here is what I have staged for commit.
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-10-06 17:33 Nathan Bossart <[email protected]>
parent: Nathan Bossart <[email protected]>
0 siblings, 0 replies; 36+ messages in thread
From: Nathan Bossart @ 2025-10-06 17:33 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Oct 03, 2025 at 03:33:21PM -0500, Nathan Bossart wrote:
> Here is what I have staged for commit.
Committed. That seems like a good stopping point for this work, so I have
marked the associated commitfest entry as "Committed."
--
nathan
^ permalink raw reply [nested|flat] 36+ messages in thread
end of thread, other threads:[~2025-10-06 17:33 UTC | newest]
Thread overview: 36+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-02-15 21:53 [PATCH v6 7/7] Update comment obsolete since 69c3936a Justin Pryzby <[email protected]>
2020-02-15 21:53 [PATCH v8 8/8] Update comment obsolete since 69c3936a Justin Pryzby <[email protected]>
2020-02-15 21:53 [PATCH v7 7/9] Update comment obsolete since 69c3936a Justin Pryzby <[email protected]>
2023-09-25 20:32 [PATCH v5 05/14] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 14:48 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-24 21:40 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-25 14:16 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-25 18:50 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-29 08:45 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-10-02 17:33 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-10-03 07:36 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-10-03 20:33 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-10-06 17:33 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox