public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v2 05/11] Improve sentences in overview of system configuration parameters
85+ messages / 12 participants
[nested] [flat]
* [PATCH v5 05/14] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_//gBZFZYGWm7urqUgDbu6PSe
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v5-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v2 05/11] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/RgO6TscyR9fMvkEm1k5N=yu
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v2-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v3 05/11] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/.arW0LC=Yi8JO9BRih4YlyS
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v3-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v2 05/11] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/RgO6TscyR9fMvkEm1k5N=yu
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v2-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v6 05/15] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/74urtnrsBSymuH7bJczNOGS
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v6-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v3 05/11] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/.arW0LC=Yi8JO9BRih4YlyS
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v3-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v7 05/16] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/VSGM3xNEmY7iJyL2wuWRCjV
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v7-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v4 05/12] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* [PATCH v4 05/12] Improve sentences in overview of system configuration parameters
@ 2023-09-25 20:32 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Karl O. Pinc @ 2023-09-25 20:32 UTC (permalink / raw)
Get rid of "we" wording. Remove extra words in sentences.
Line break after end of each sentence to ease future patch reading.
---
doc/src/sgml/config.sgml | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6bc1b215db..97f9838bfb 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -10,9 +10,10 @@
<para>
There are many configuration parameters that affect the behavior of
- the database system. In the first section of this chapter we
- describe how to interact with configuration parameters. The subsequent sections
- discuss each parameter in detail.
+ the database system.
+ The first section of this chapter describes how to interact with
+ configuration parameters.
+ Subsequent sections discuss each parameter in detail.
</para>
<sect1 id="config-setting">
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0006-Provide-examples-of-listing-all-settings.patch
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
@ 2025-01-16 01:27 Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Tom Lane @ 2025-01-16 01:27 UTC (permalink / raw)
To: David Rowley <[email protected]>; +Cc: John Naylor <[email protected]>; Nathan Bossart <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
David Rowley <[email protected]> writes:
> I agree that the evidence you (John) gathered is enough reason to use memcpy().
Okay ... doesn't quite match my intuition, but intuition is a poor
guide to such things.
regards, tom lane
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
@ 2025-01-17 17:15 ` Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-01-17 17:15 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: David Rowley <[email protected]>; John Naylor <[email protected]>; [email protected] <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
With commit e24d770 in place, I took a closer look at hex_decode(), and I
concluded that doing anything better without intrinsics would likely
require either a huge lookup table or something with complexity rivalling
the instrinsics approach (while also not rivalling its performance). So, I
took a closer look at the instrinsics patches and had the following
thoughts:
* The approach looks generally reasonable to me, but IMHO the code needs
much more commentary to explain how it works.
* The functions that test the length before potentially calling a function
pointer should probably be inlined (see pg_popcount() in pg_bitutils.h).
I wouldn't be surprised if some compilers are inlining this stuff
already, but it's probably worth being explicit about it.
* Finally, I think we should ensure we've established a really strong case
for this optimization. IME these intrinsics patches require a ton of
time and energy, and the code is often extremely complex. I would be
interested to see how your bytea test compares with the improvements
added in commit e24d770 and with sending the data in binary.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-01-22 10:58 ` [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: [email protected] @ 2025-01-22 10:58 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> The approach looks generally reasonable to me, but IMHO the code needs
much more commentary to explain how it works.
Added comments to explain the SVE implementation.
> I would be interested to see how your bytea test compares with the
improvements added in commit e24d770 and with sending the data in binary.
The following are the bytea test results with commit e24d770.
The same query and tables were used.
With commit e24d770:
Query exec time: 2.324 sec
hex_encode function time: 0.72 sec
Pre-commit e24d770:
Query exec time: 2.858 sec
hex_encode function time: 1.228 sec
SVE patch:
Query exec time: 1.654 sec
hex_encode_sve function time: 0.085 sec
> The functions that test the length before potentially calling a function
> pointer should probably be inlined (see pg_popcount() in pg_bitutils.h).
> I wouldn't be surprised if some compilers are inlining this stuff
> already, but it's probably worth being explicit about it.
Should we implement an inline function in "utils/builtins.h", similar to
pg_popcount()? Currently, we have not modified the header file, everything
is statically implemented in encode.c.
---
Chiranmoy
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-01-22 11:10 ` [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-01-22 11:10 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
I realized I didn't attach the patch.
Attachments:
[application/octet-stream] v2-0001-SVE-support-for-hex-encode-and-hex-decode.patch (15.8K, ../../OSBPR01MB266437C48E1F5ACB3D8C69C197E12@OSBPR01MB2664.jpnprd01.prod.outlook.com/3-v2-0001-SVE-support-for-hex-encode-and-hex-decode.patch)
download | inline diff:
From 2094bc7f60db93693f2c054e9044d8baa128bb8f Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Wed, 22 Jan 2025 15:52:40 +0530
Subject: [PATCH v2] SVE support for hex encode and hex decode
---
config/c-compiler.m4 | 53 ++++++++
configure | 63 +++++++++
configure.ac | 9 ++
meson.build | 47 +++++++
src/backend/utils/adt/encode.c | 241 +++++++++++++++++++++++++++++++++
src/include/pg_config.h.in | 3 +
6 files changed, 416 insertions(+)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 8534cc54c1..bb22ceed17 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -704,3 +704,56 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_AVX512_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the ARM SVE intrinsic required for hex coding:
+# svld1, svtbl, svsel, etc.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[
+ AC_CACHE_CHECK([for svtbl, svlsr_z, svand_z, svcreate2, svst2, svsel and svget2 intrinsics],
+ [pgac_cv_arm_sve_hex_intrinsics],
+ [
+
+ AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>],
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+
+ [
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_z(pred, bytes, 4);
+ bytes = svand_z(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, bytes);
+ cmp1 = svcmplt(pred, bytes, 0);
+ cmp2 = svcmpgt(pred, bytes, 0);
+ bytes = svsel(svnot_z(pred, svand_z(pred, cmp1, cmp2)), bytes, bytes);
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ ])],
+ [pgac_cv_arm_sve_hex_intrinsics=yes],
+ [pgac_cv_arm_sve_hex_intrinsics=no])
+
+ ])
+
+ if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics = yes
+ fi
+])
diff --git a/configure b/configure
index ceeef9b091..e634feec02 100755
--- a/configure
+++ b/configure
@@ -17168,6 +17168,69 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for SVE intrinsic svtbl, svlsr_z, etc." >&5
+ $as_echo_n "checking for SVE intrinsic svtbl, svlsr_z... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute(target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int
+main ()
+{
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_z(pred, bytes, 4);
+ bytes = svand_z(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, bytes);
+ cmp1 = svcmplt(pred, bytes, 0);
+ cmp2 = svcmpgt(pred, bytes, 0);
+ bytes = svsel(svnot_z(pred, svand_z(pred, cmp1, cmp2)), bytes, bytes);
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS=yes
+fi
+
+if test x"$PGAC_ARM_SVE_HEX_INTRINSICS" = x"yes"; then
+ $as_echo "#define USE_SVE_WITH_RUNTIME_CHECK 1" >>confdefs.h
+fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index d713360f34..cc805667b9 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2021,6 +2021,15 @@ if test x"$host_cpu" = x"x86_64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$PGAC_ARM_SVE_HEX_INTRINSICS" = x"yes"; then
+ AC_DEFINE(USE_SVE_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index 32fc89f3a4..d9d13b3c55 100644
--- a/meson.build
+++ b/meson.build
@@ -2194,6 +2194,53 @@ int main(void)
endif
+###############################################################
+# Check the availability of ARM SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_z(pred, bytes, 4);
+ bytes = svand_z(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, bytes);
+ cmp1 = svcmplt(pred, bytes, 0);
+ cmp2 = svcmpgt(pred, bytes, 0);
+ bytes = svsel(svnot_z(pred, svand_z(pred, cmp1, cmp2)), bytes, bytes);
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+}
+'''
+
+ if cc.links(prog, name: 'ARM SVE hex encoding', args: test_c_args)
+ cdata.set('USE_SVE_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d..0fe41a8d00 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -20,6 +20,10 @@
#include "utils/memutils.h"
#include "varatt.h"
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+#include <sys/auxv.h>
+#include <arm_sve.h>
+#endif
/*
* Encoding conversion API.
@@ -177,8 +181,106 @@ static const int8 hexlookup[128] = {
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
};
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+static uint64 hex_encode_slow(const char *src, size_t len, char *dst);
+static uint64 hex_decode_slow(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_slow(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst,
+ Node *escontext);
+uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext) =
+ hex_decode_safe_choose;
+
+/*
+ * Returns true if the CPU supports SVE instructions.
+ */
+static inline bool
+check_sve_support(void)
+{
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_slow;
+ hex_decode_optimized = hex_decode_slow;
+ hex_decode_safe_optimized = hex_decode_safe_slow;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+ if (len < 16)
+ return hex_encode_slow(src, len, dst);
+ return hex_encode_optimized(src, len, dst);
+}
+
+uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+ if (len < 32)
+ return hex_decode_slow(src, len, dst);
+ return hex_decode_optimized(src, len, dst);
+}
+
+uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+ if (len < 32)
+ return hex_decode_safe_slow(src, len, dst, escontext);
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+uint64
+hex_encode_slow(const char *src, size_t len, char *dst)
+#else
uint64
hex_encode(const char *src, size_t len, char *dst)
+#endif
{
const char *end = src + len;
@@ -207,14 +309,24 @@ get_hex(const char *cp, char *out)
return (res >= 0);
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+uint64
+hex_decode_slow(const char *src, size_t len, char *dst)
+#else
uint64
hex_decode(const char *src, size_t len, char *dst)
+#endif
{
return hex_decode_safe(src, len, dst, NULL);
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+uint64
+hex_decode_safe_slow(const char *src, size_t len, char *dst, Node *escontext)
+#else
uint64
hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+#endif
{
const char *s,
*srcend;
@@ -254,6 +366,135 @@ hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
return p - dst;
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+/*
+ * SVE implementation of hex_encode and hex_decode.
+ */
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ svbool_t pred;
+ svuint8_t bytes,
+ high,
+ low,
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svuint8x2_t merged;
+ uint32 vec_len = svcntb();
+
+ for (size_t i = 0; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8(i, len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_z(pred, bytes, 4); /* shift-right to get the high nibble */
+ low = svand_z(pred, bytes, 0xF); /* mask high to get the low nibble */
+
+ /*
+ * Convert the nibbles to hex digits by indexing into hextbl_vec,
+ * for example, a nibble value of 10 indexed into hextbl_vec gives 'a'.
+ * Finally, interleave the high and low nibbles
+ */
+ merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, merged);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII values '0'-'9' to integers 0-9 by subtracting 48.
+ * Similarly, convert letters 'A'-'F' and 'a'-'f' to integers 10-15.
+ */
+ svuint8_t dgt_vec = svsub_x(pred, vec, 48),
+ cap_vec = svsub_x(pred, vec, 55),
+ sml_vec = svsub_x(pred, vec, 87),
+ letter_vec;
+ /*
+ * Identify valid integers in dgt_vec, cap_vec, and sml_vec.
+ * Values 0-9 are valid in dgt_vec, while values 10-15 are valid
+ * in cap_vec and sml_vec.
+ */
+ svbool_t dgt_bool = svcmplt(pred, dgt_vec, 10),
+ cap_bool = svcmplt(pred, cap_vec, 16),
+ letter_bool;
+ /*
+ * Combine cap_vec and sml_vec and mark the valid range 10-15.
+ */
+ letter_vec = svsel(cap_bool, cap_vec, sml_vec);
+ letter_bool = svand_z(pred, svcmpgt(pred, letter_vec, 9),
+ svcmplt(pred, letter_vec, 16));
+ /*
+ * Check for invalid hexadecimal digits. Each value must fall
+ * within the range 0-9 (true in dgt_bool) or 10-15 (true in letter_bool).
+ */
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, dgt_bool, letter_bool))))
+ return false;
+
+ /* Finally, combine dgt_vec and letter_vec */
+ *res = svsel(dgt_bool, dgt_vec, letter_vec);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ svbool_t pred;
+ svuint8x2_t bytes;
+ svuint8_t high,
+ low;
+ uint32 processed;
+ size_t i = 0,
+ loop_bytes = len & ~1; /* handles inputs of odd length */
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ pred = svwhilelt_b8(i / 2, len / 2);
+ bytes = svld2(pred, (uint8 *) src);
+ high = svget2(bytes, 0); /* hex digit for high nibble */
+ low = svget2(bytes, 1); /* hex digit for low nibble */
+
+ /* fall back if ASCII less than '0' is found */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'),
+ svcmplt(pred, low, '0'))))
+ break;
+
+ /* fall back if invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* left-shift high and perform bitwise OR with low to form the byte */
+ svst1(pred, (uint8 *) dst, svorr_x(pred, svlsl_x(pred, high, 4), low));
+
+ processed = svcntp_b8(pred, pred) * 2;
+ src += processed;
+ i += processed;
+ dst += processed / 2;
+ }
+
+ if (i < len) /* fall back */
+ return dst - p + hex_decode_safe_slow(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
static uint64
hex_enc_len(const char *src, size_t srclen)
{
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798ab..b5096c11f4 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -648,6 +648,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-01-25 22:07 ` Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-01-25 22:07 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Jan 22, 2025 at 11:10:10AM +0000, [email protected] wrote:
> I realized I didn't attach the patch.
Thanks. Would you mind creating a commitfest entry for this one?
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-02-04 06:06 ` [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-02-04 06:06 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Inlined the hex encode/decode functions in "src/include/utils/builtins.h"
similar to pg_popcount() in pg_bitutils.h.
---
Chiranmoy
Attachments:
[application/octet-stream] v3-0001-SVE-support-for-hex-encode-and-hex-decode.patch (18.3K, ../../TY2PR01MB26671195A71369185F7BD08C97F42@TY2PR01MB2667.jpnprd01.prod.outlook.com/3-v3-0001-SVE-support-for-hex-encode-and-hex-decode.patch)
download | inline diff:
From 015afdfa5b1eccc039bc1c276dd7a51d3729257a Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Tue, 4 Feb 2025 11:26:41 +0530
Subject: [PATCH v3] SVE support for hex encode and hex decode
---
config/c-compiler.m4 | 58 +++++++++
configure | 79 ++++++++++++
configure.ac | 9 ++
meson.build | 56 +++++++++
src/backend/utils/adt/encode.c | 212 ++++++++++++++++++++++++++++++++-
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 55 ++++++++-
7 files changed, 466 insertions(+), 6 deletions(-)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 8534cc54c1..d99ecfb2a7 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -704,3 +704,61 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_AVX512_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the ARM SVE intrinsic required for hex coding:
+# svtbl, svlsr_x, svand_z, svcreate2, etc.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input@<:@32@:>@;
+ char output@<:@32@:>@;
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@ == 0;
+ }
+
+ return 0;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index ceeef9b091..e445cb1451 100755
--- a/configure
+++ b/configure
@@ -17168,6 +17168,85 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_z, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_z, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input[32];
+ char output[32];
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index d713360f34..2dbb678cae 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2021,6 +2021,15 @@ if test x"$host_cpu" = x"x86_64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index 8e128f4982..6a10331acf 100644
--- a/meson.build
+++ b/meson.build
@@ -2194,6 +2194,62 @@ int main(void)
endif
+###############################################################
+# Check the availability of ARM SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int vec_len = svcntb();
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+}
+'''
+
+ if cc.links(prog, name: 'ARM SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d..cf0137a1f1 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -20,6 +20,12 @@
#include "utils/memutils.h"
#include "varatt.h"
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+#endif
/*
* Encoding conversion API.
@@ -177,8 +183,81 @@ static const int8 hexlookup[128] = {
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
};
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst,
+ Node *escontext);
+uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext) =
+ hex_decode_safe_choose;
+
+/*
+ * Returns true if the CPU supports SVE instructions.
+ */
+static inline bool
+check_sve_support(void)
+{
+#if defined(HAVE_ELF_AUX_INFO) && defined(__aarch64__) /* FreeBSD */
+ unsigned long value;
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL) && defined(__aarch64__) /* Linux */
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +287,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
@@ -254,6 +333,133 @@ hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
return p - dst;
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+/*
+ * SVE implementation of hex_encode and hex_decode.
+ */
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ svbool_t pred;
+ svuint8_t bytes,
+ high,
+ low,
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svuint8x2_t merged;
+ uint32 vec_len = svcntb();
+
+ for (size_t i = 0; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8(i, len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4); /* shift-right to get the high nibble */
+ low = svand_z(pred, bytes, 0xF); /* mask high to get the low nibble */
+
+ /*
+ * Convert the nibbles to hex digits by indexing into hextbl_vec,
+ * for example, a nibble value of 10 indexed into hextbl_vec gives 'a'.
+ * Finally, interleave the high and low nibbles.
+ */
+ merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, merged);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII values '0'-'9' to integers 0-9 by subtracting 48.
+ * Similarly, convert letters 'A'-'F' and 'a'-'f' to integers 10-15.
+ */
+ svuint8_t dgt_vec = svsub_x(pred, vec, 48),
+ cap_vec = svsub_x(pred, vec, 55),
+ sml_vec = svsub_x(pred, vec, 87),
+ ltr_vec;
+ /*
+ * Identify valid integers in dgt_vec, cap_vec, and sml_vec.
+ * Integers 0-9 are valid in dgt_vec, while integers 10-15 are valid
+ * in cap_vec and sml_vec.
+ */
+ svbool_t valid_dgt = svcmplt(pred, dgt_vec, 10),
+ valid_ltr;
+
+ /* Combine cap_vec and sml_vec and mark the valid range 10-15. */
+ ltr_vec = svsel(svcmplt(pred, cap_vec, 16), cap_vec, sml_vec);
+ valid_ltr = svand_z(pred, svcmpgt(pred, ltr_vec, 9),
+ svcmplt(pred, ltr_vec, 16));
+ /*
+ * Check for invalid hexadecimal digits. Each value must fall
+ * within the range 0-9 (true in valid_dgt) or 10-15 (true in valid_ltr).
+ */
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_dgt, valid_ltr))))
+ return false;
+
+ /* Finally, combine dgt_vec and ltr_vec */
+ *res = svsel(valid_dgt, dgt_vec, ltr_vec);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ svbool_t pred;
+ svuint8x2_t bytes;
+ svuint8_t high,
+ low;
+ uint32 processed;
+ size_t i = 0,
+ loop_bytes = len & ~1; /* handles inputs of odd length */
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ pred = svwhilelt_b8(i / 2, len / 2);
+ bytes = svld2(pred, (uint8 *) src);
+ high = svget2(bytes, 0); /* hex digits for high nibble */
+ low = svget2(bytes, 1); /* hex digits for low nibble */
+
+ /* fallback if a character below ASCII '0' is found. */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'),
+ svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* left-shift high and perform bitwise OR with low to form the byte */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ processed = svcntp_b8(pred, pred) * 2;
+ src += processed;
+ i += processed;
+ dst += processed / 2;
+ }
+
+ if (i < len) /* fall back */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
static uint64
hex_enc_len(const char *src, size_t srclen)
{
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798ab..b5096c11f4 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -648,6 +648,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d225..e9b1f963dd 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,60 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * We can use SVE intrinsics for hex-coding, but only if we can
+ * verify that the CPU supports it via a runtime check.
+ */
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext);
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-02-19 08:19 ` [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-02-19 08:19 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
It seems that the patch doesn't compile on macOS, it is unable to map 'i'
and 'len' which are of type 'size_t' to 'uint64'. This appears to be a mac specific
issue. The latest patch should resolve this by casting 'size_t' to 'uint64' before
passing it to 'svwhilelt_b8'.
[11:04:07.478] ../src/backend/utils/adt/encode.c:356:10: error: call to 'svwhilelt_b8' is ambiguous
[11:04:07.478] 356 | pred = svwhilelt_b8(i, len);
[11:04:07.478] | ^~~~~~~~~~~~
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28288:10: note: candidate function
[11:04:07.478] 28288 | svbool_t svwhilelt_b8(uint32_t, uint32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28296:10: note: candidate function
[11:04:07.478] 28296 | svbool_t svwhilelt_b8(uint64_t, uint64_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28304:10: note: candidate function
[11:04:07.478] 28304 | svbool_t svwhilelt_b8(int32_t, int32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28312:10: note: candidate function
[11:04:07.478] 28312 | svbool_t svwhilelt_b8(int64_t, int64_t);
[11:04:07.478] | ^
[11:04:07.478] ../src/backend/utils/adt/encode.c:433:10: error: call to 'svwhilelt_b8' is ambiguous
[11:04:07.478] 433 | pred = svwhilelt_b8(i / 2, len / 2);
[11:04:07.478] | ^~~~~~~~~~~~
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28288:10: note: candidate function
[11:04:07.478] 28288 | svbool_t svwhilelt_b8(uint32_t, uint32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28296:10: note: candidate function
[11:04:07.478] 28296 | svbool_t svwhilelt_b8(uint64_t, uint64_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28304:10: note: candidate function
[11:04:07.478] 28304 | svbool_t svwhilelt_b8(int32_t, int32_t);
[11:04:07.478] | ^
[11:04:07.478] /Applications/Xcode_16.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/include/arm_sve.h:28312:10: note: candidate function
[11:04:07.478] 28312 | svbool_t svwhilelt_b8(int64_t, int64_t);
[11:04:07.478] | ^
[11:04:07.478] 2 errors generated.
---
Chiranmoy
Attachments:
[application/octet-stream] v4-0001-SVE-support-for-hex-encode-and-hex-decode.patch (18.4K, ../../OSBPR01MB2664ED032A81FBF21BEC82EE97C52@OSBPR01MB2664.jpnprd01.prod.outlook.com/3-v4-0001-SVE-support-for-hex-encode-and-hex-decode.patch)
download | inline diff:
From 015afdfa5b1eccc039bc1c276dd7a51d3729257a Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Tue, 4 Feb 2025 11:26:41 +0530
Subject: [PATCH v3] SVE support for hex encode and hex decode
---
config/c-compiler.m4 | 58 +++++++++
configure | 79 ++++++++++++
configure.ac | 9 ++
meson.build | 56 +++++++++
src/backend/utils/adt/encode.c | 212 ++++++++++++++++++++++++++++++++-
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 55 ++++++++-
7 files changed, 466 insertions(+), 6 deletions(-)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 8534cc54c1..d99ecfb2a7 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -704,3 +704,61 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_AVX512_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the ARM SVE intrinsic required for hex coding:
+# svtbl, svlsr_x, svand_z, svcreate2, etc.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input@<:@32@:>@;
+ char output@<:@32@:>@;
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@ == 0;
+ }
+
+ return 0;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index ceeef9b091..e445cb1451 100755
--- a/configure
+++ b/configure
@@ -17168,6 +17168,85 @@ $as_echo "#define USE_AVX512_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_z, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_z, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int vec_len = svcntb();
+ char input[32];
+ char output[32];
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index d713360f34..2dbb678cae 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2021,6 +2021,15 @@ if test x"$host_cpu" = x"x86_64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index 8e128f4982..6a10331acf 100644
--- a/meson.build
+++ b/meson.build
@@ -2194,6 +2194,62 @@ int main(void)
endif
+###############################################################
+# Check the availability of ARM SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int vec_len = svcntb();
+ char input[64] = {0};
+ char output[64] = {0};
+ svbool_t pred = svptrue_b8(), cmp1, cmp2;
+ svuint8_t bytes, hextbl_vec;
+ svuint8x2_t merged;
+
+ if (vec_len >= 16)
+ {
+ /* intrinsics used in hex_encode_sve */
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) "0123456789ABCDEF");
+ bytes = svld1(pred, (uint8_t *) input);
+ bytes = svlsr_x(pred, bytes, 4);
+ bytes = svand_x(pred, bytes, 0xF);
+ merged = svcreate2(svtbl(hextbl_vec, bytes), svtbl(hextbl_vec, bytes));
+ svst2(pred, (uint8_t *) output, merged);
+
+ /* intrinsics used in hex_decode_sve */
+ bytes = svget2(svld2(pred, (uint8_t *) output), 0);
+ bytes = svsub_x(pred, bytes, 48);
+ cmp1 = svcmplt(pred, bytes, 16);
+ cmp2 = svcmpgt(pred, bytes, 9);
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, cmp1, cmp2))))
+ return 0;
+ bytes = svsel(svand_z(pred, cmp1, cmp2), bytes, bytes);
+ bytes = svlsl_x(pred, bytes, svcntp_b8(pred, pred));
+ svst1(pred, output, bytes);
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0] == 0;
+ }
+
+ return 0;
+}
+'''
+
+ if cc.links(prog, name: 'ARM SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d..cf0137a1f1 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -20,6 +20,12 @@
#include "utils/memutils.h"
#include "varatt.h"
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+#endif
/*
* Encoding conversion API.
@@ -177,8 +183,81 @@ static const int8 hexlookup[128] = {
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
};
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst,
+ Node *escontext);
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst,
+ Node *escontext);
+uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext) =
+ hex_decode_safe_choose;
+
+/*
+ * Returns true if the CPU supports SVE instructions.
+ */
+static inline bool
+check_sve_support(void)
+{
+#if defined(HAVE_ELF_AUX_INFO) && defined(__aarch64__) /* FreeBSD */
+ unsigned long value;
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL) && defined(__aarch64__) /* Linux */
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +287,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
@@ -254,6 +333,133 @@ hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
return p - dst;
}
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+/*
+ * SVE implementation of hex_encode and hex_decode.
+ */
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ svbool_t pred;
+ svuint8_t bytes,
+ high,
+ low,
+ hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svuint8x2_t merged;
+ uint32 vec_len = svcntb();
+
+ for (size_t i = 0; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8((uint64) i, (uint64) len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4); /* shift-right to get the high nibble */
+ low = svand_z(pred, bytes, 0xF); /* mask high to get the low nibble */
+
+ /*
+ * Convert the nibbles to hex digits by indexing into hextbl_vec,
+ * for example, a nibble value of 10 indexed into hextbl_vec gives 'a'.
+ * Finally, interleave the high and low nibbles.
+ */
+ merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, merged);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII values '0'-'9' to integers 0-9 by subtracting 48.
+ * Similarly, convert letters 'A'-'F' and 'a'-'f' to integers 10-15.
+ */
+ svuint8_t dgt_vec = svsub_x(pred, vec, 48),
+ cap_vec = svsub_x(pred, vec, 55),
+ sml_vec = svsub_x(pred, vec, 87),
+ ltr_vec;
+ /*
+ * Identify valid integers in dgt_vec, cap_vec, and sml_vec.
+ * Integers 0-9 are valid in dgt_vec, while integers 10-15 are valid
+ * in cap_vec and sml_vec.
+ */
+ svbool_t valid_dgt = svcmplt(pred, dgt_vec, 10),
+ valid_ltr;
+
+ /* Combine cap_vec and sml_vec and mark the valid range 10-15. */
+ ltr_vec = svsel(svcmplt(pred, cap_vec, 16), cap_vec, sml_vec);
+ valid_ltr = svand_z(pred, svcmpgt(pred, ltr_vec, 9),
+ svcmplt(pred, ltr_vec, 16));
+ /*
+ * Check for invalid hexadecimal digits. Each value must fall
+ * within the range 0-9 (true in valid_dgt) or 10-15 (true in valid_ltr).
+ */
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_dgt, valid_ltr))))
+ return false;
+
+ /* Finally, combine dgt_vec and ltr_vec */
+ *res = svsel(valid_dgt, dgt_vec, ltr_vec);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ svbool_t pred;
+ svuint8x2_t bytes;
+ svuint8_t high,
+ low;
+ uint32 processed;
+ size_t i = 0,
+ loop_bytes = len & ~1; /* handles inputs of odd length */
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ pred = svwhilelt_b8((uint64) i / 2, (uint64) len / 2);
+ bytes = svld2(pred, (uint8 *) src);
+ high = svget2(bytes, 0); /* hex digits for high nibble */
+ low = svget2(bytes, 1); /* hex digits for low nibble */
+
+ /* fallback if a character below ASCII '0' is found. */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'),
+ svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* left-shift high and perform bitwise OR with low to form the byte */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ processed = svcntp_b8(pred, pred) * 2;
+ src += processed;
+ i += processed;
+ dst += processed / 2;
+ }
+
+ if (i < len) /* fall back */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
static uint64
hex_enc_len(const char *src, size_t srclen)
{
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 07b2f798ab..b5096c11f4 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -648,6 +648,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d225..e9b1f963dd 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,60 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * We can use SVE intrinsics for hex-coding, but only if we can
+ * verify that the CPU supports it via a runtime check.
+ */
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized)
+ (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized)
+ (const char *src, size_t len, char *dst, Node *escontext);
+#endif /* USE_SVE_WITH_RUNTIME_CHECK */
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-06-04 13:47 ` Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-06-04 13:47 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
I have marked the commitfest entry for this [0] as waiting-on-author
because the patch needs to be rebased.
[0] https://commitfest.postgresql.org/patch/5538/
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-06-09 09:49 ` [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-06-09 09:49 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Here's the rebased patch with a few modifications.
The hand-unrolled hex encode performs better than the non-unrolled version on
r8g.4xlarge. No improvement on m7g.4xlarge.
Added line-by-line comments explaining the changes with an example.
Below are the results. Input size is in bytes, and exec time is in ms.
encode - r8g.4xlarge
Input | master | SVE | SVE-unrolled
-------+--------+--------+--------------
8 | 4.971 | 6.434 | 6.623
16 | 8.532 | 4.399 | 4.710
24 | 12.296 | 5.007 | 5.780
32 | 16.003 | 5.027 | 5.234
40 | 19.628 | 5.807 | 6.201
48 | 23.277 | 5.815 | 6.222
56 | 26.927 | 6.744 | 7.030
64 | 30.419 | 6.774 | 6.347
128 | 83.250 | 10.214 | 9.104
256 |112.158 | 17.892 | 16.313
512 |216.544 | 31.060 | 29.876
1024 |429.351 | 59.310 | 53.374
2048 |854.677 |116.769 | 101.004
4096 |1706.528|237.322 | 195.297
8192 |3723.884|499.520 | 385.424
---------------------------------------
encode - m7g.4xlarge
Input | master | SVE | SVE-unrolled
-------+--------+--------+--------------
8 | 5.503 | 7.986 | 8.053
16 | 9.881 | 9.583 | 9.888
24 | 13.854 | 9.212 | 10.138
32 | 18.056 | 9.208 | 9.364
40 | 22.127 | 10.134 | 10.540
48 | 26.214 | 10.186 | 10.550
56 | 29.718 | 10.197 | 10.428
64 | 33.613 | 10.982 | 10.497
128 | 66.060 | 12.460 | 12.624
256 |130.225 | 18.491 | 18.872
512 |267.105 | 30.343 | 31.661
1024 |515.603 | 54.371 | 55.341
2048 |1013.766|103.898 | 105.192
4096 |2018.705|202.653 | 203.142
8192 |4000.496|400.918 | 401.842
---------------------------------------
decode - r8g.4xlarge
Input | master | SVE
-------+--------+--------
8 | 7.641 | 8.787
16 | 14.301 | 14.477
32 | 28.663 | 6.091
48 | 42.940 | 17.604
64 | 57.483 | 10.549
80 | 71.637 | 19.194
96 | 85.918 | 15.586
112 |100.272 | 25.956
128 |114.740 | 19.829
256 |229.176 | 36.032
512 |458.295 | 68.222
1024 |916.741 |132.927
2048 |1833.422|262.741
4096 |3667.096|522.009
8192 |7333.886|1042.447
---------------------------------------
decode - m7g.4xlarge
Input | master | SVE
-------+--------+--------
8 | 8.194 | 9.433
16 | 14.397 | 15.606
32 | 26.669 | 29.006
48 | 45.971 | 48.984
64 | 58.468 | 12.388
80 | 70.820 | 22.295
96 | 84.792 | 43.470
112 | 98.992 | 54.282
128 |113.250 | 25.508
256 |218.743 | 45.165
512 |414.133 | 86.800
1024 |828.493 |174.670
2048 |1617.921|346.375
4096 |3259.159|689.391
8192 |6551.879|1376.195
--------
Chiranmoy
Attachments:
[application/octet-stream] v5-0001-SVE-support-for-hex-coding.patch (24.6K, ../../OS9PR01MB15185A8BA8E15E4AFD8B14490976BA@OS9PR01MB15185.jpnprd01.prod.outlook.com/3-v5-0001-SVE-support-for-hex-coding.patch)
download | inline diff:
From 3a508684171ae411e4e8251c717b61a8def04c1f Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Mon, 9 Jun 2025 14:16:26 +0530
Subject: [PATCH v5] SVE support for hex coding
---
config/c-compiler.m4 | 85 ++++++++
configure | 104 +++++++++
configure.ac | 9 +
meson.build | 81 +++++++
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/encode.c | 6 +-
src/backend/utils/adt/encode_aarch64.c | 278 +++++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 51 ++++-
10 files changed, 613 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/adt/encode_aarch64.c
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index 5f3e1d1faf9..20e71cd8546 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -797,3 +797,88 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_SVE_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the SVE intrinsic required for hex coding:
+# svsub_x, svcmplt, svsel, svcmpgt, svtbl, svlsr_x, svand_z, svcreate2,
+# svptest_any, svnot_z, svorr_z, svcntb, svld1, svwhilelt_b8, svst2, svld2,
+# svget2, svst1 and svlsl_x.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+
+ char input@<:@64@:>@;
+ char output@<:@128@:>@;
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index 4f15347cc95..4d5d6acefb5 100755
--- a/configure
+++ b/configure
@@ -17851,6 +17851,110 @@ $as_echo "#define USE_SVE_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_x, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_x, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+
+ char input[64];
+ char output[128];
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_HEX_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index 4b8335dc613..fcae9b84616 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2107,6 +2107,15 @@ if test x"$host_cpu" = x"aarch64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_HEX_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index d142e3e408b..de2d1ebd384 100644
--- a/meson.build
+++ b/meson.build
@@ -2384,6 +2384,87 @@ int main(void)
endif
+###############################################################
+# Check the availability of SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+
+char input[64];
+char output[128];
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+}
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char hextbl[] = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+}
+'''
+
+ if cc.links(prog, name: 'SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_HEX_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index 4a233b63c32..2a3ba1d4485 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -32,6 +32,7 @@ OBJS = \
dbsize.o \
domains.o \
encode.o \
+ encode_aarch64.o \
enum.o \
expandeddatum.o \
expandedrecord.o \
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d1..fa62ce3107d 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -178,7 +178,7 @@ static const int8 hexlookup[128] = {
};
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +208,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
new file mode 100644
index 00000000000..574a7550469
--- /dev/null
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -0,0 +1,278 @@
+/*-------------------------------------------------------------------------
+ *
+ * encode_aarch64.c
+ * Holds the SVE hex encode/decode implementations.
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/encode_aarch64.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <c.h>
+
+#include "utils/builtins.h"
+
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+
+/*
+ * These are the SVE implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * The function pointers are initially set to "choose" functions. These
+ * functions will first set the pointers to the right implementations (based on
+ * what the current CPU supports) and then will call the pointer to fulfill the
+ * caller's request.
+ */
+
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext);
+uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext) = hex_decode_safe_choose;
+
+static inline bool
+check_sve_support(void)
+{
+#ifdef HAVE_ELF_AUX_INFO
+ unsigned long value;
+
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL)
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint32 vec_len = svcntb();
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svbool_t pred = svptrue_b8();
+ size_t loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svuint8_t bytes, high, low;
+ svuint8x2_t zipped;
+
+ for (size_t i = 0; i < loop_bytes; i += 2 * vec_len)
+ {
+ bytes = svld1(pred, (uint8 *) src);
+
+ /* Right-shift to obtain the high nibble */
+ high = svlsr_x(pred, bytes, 4);
+
+ /* Mask the high nibble to obtain the low nibble */
+ low = svand_z(pred, bytes, 0xF);
+
+ /*
+ * Convert the high and low nibbles to hexadecimal digits using a
+ * vectorized table lookup and zip (interleave) the hexadecimal digits.
+ */
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+
+ /* unrolled */
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ /* process remaining tail bytes */
+ for (size_t i = loop_bytes; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8((uint64) i, (uint64) len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+/*
+ * get_hex_sve
+ * Returns true if the hexadecimal digits are successfully converted
+ * to nibbles and stored in 'res'; otherwise, returns false.
+ */
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII of '0'-'9' to integers 0-9 by subtracting 48 (ASCII of '0').
+ * Similarly, convert letters 'A'–'F' and 'a'–'f' to integers 10–15 by
+ * subtracting 55 ('A' - 10) and 87 ('a' - 10).
+ */
+ svuint8_t digit = svsub_x(pred, vec, '0'),
+ upper = svsub_x(pred, vec, 'A' - 10),
+ lower = svsub_x(pred, vec, 'a' - 10);
+
+ /*
+ * Identify valid values in digits, upper, and lower vectors.
+ * Values 0-9 are valid in digits, while values 10-15 are valid
+ * in upper and lower.
+ *
+ * Example:
+ * vec: '0' '9' 'A' 'F' 'a' 'f'
+ * vec (in ASCII): 48 57 65 70 97 102
+ *
+ * digit: 0 9 17 22 49 54
+ * valid_digit: 1 1 0 0 0 0
+ *
+ * upper: 249 2 10 15 42 47
+ * valid_upper: 0 1 1 1 0 0
+ *
+ * lower: 217 226 234 239 10 15
+ *
+ * Note that values 0-9 are also marked valid in valid_upper, this will be
+ * handled later.
+ */
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+
+ /*
+ * Merge upper and lower vector using the logic: take the element from
+ * upper if it's true in valid_upper else pick the element in lower
+ *
+ * Mark the valid range i.e. 10-15 in letter vector
+ *
+ * letter: 217 2 10 15 10 15
+ * valid_letter: 0 0 1 1 1 1
+ */
+
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+
+ /*
+ * Check for invalid hexadecimal digit. Each value must fall within
+ * the range 0-9 (true in valid_digit) or 10-15 (true in valid_letter) i.e.
+ * the OR of valid_digit and valid_letter should be all true.
+ */
+
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return false;
+
+ /*
+ * Finally, combine digit and letter vectors using the logic:
+ * take the element from digit if it's true in valid_digit else pick the
+ * element in letter.
+ *
+ * res: 0 9 10 15 10 15
+ */
+
+ *res = svsel(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = svcntb();
+ size_t loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svbool_t pred = svptrue_b8();
+ const char *p = dst;
+
+ for (size_t i = 0; i < loop_bytes; i += 2 * vec_len)
+ {
+ svuint8x2_t bytes = svld2(pred, (uint8 *) src);
+ svuint8_t high = svget2(bytes, 0),
+ low = svget2(bytes, 1);
+
+ /* fallback for characters with ASCII values below '0' */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if an invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* form the byte by left-shifting the high nibble and OR-ing it with the low nibble */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > loop_bytes) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - loop_bytes, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* USE_SVE_HEX_WITH_RUNTIME_CHECK */
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index 244f48f4fd7..ea88dd77390 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -21,6 +21,7 @@ backend_sources += files(
'dbsize.c',
'domains.c',
'encode.c',
+ 'encode_aarch64.c',
'enum.c',
'expandeddatum.c',
'expandedrecord.c',
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 726a7c1be1f..7a227f1875f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -675,6 +675,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_HEX_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d2255..2f72d8df9d1 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,56 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * On AArch64, we can try to use an SVE optimized hex encode/decode on some systems.
+ */
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext);
+#endif
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-07-07 10:41 ` [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-07-07 10:41 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Attaching the rebased patch, some regression tests for SIMD hex-coding,
and a script to test bytea performance (usage info in the script).
The results obtained using the script on an m7g.4xlarge are shown below.
Read Operation
table (MB) | HEAD (ms) | SVE (ms) | improvement (%)
---------------------------------------------------
52 | 136 | 111 | 18.38
105 | 215 | 164 | 23.72
209 | 452 | 331 | 26.76
419 | 830 | 602 | 27.46
Write Operation - table size after write
table (MB) | HEAD (ms) | SVE (ms) | improvement (%)
---------------------------------------------------
52 | 1430 | 1361 | 4.82
105 | 2956 | 2816 | 4.73
The bytea write numbers are averaged over 7 runs, with the table
truncated and vacuumed after each run.
--------
Chiranmoy
Attachments:
[text/x-python] bytea_test.py (2.4K, ../../OS9PR01MB15185626CA7C6EAF82C93ABA3974FA@OS9PR01MB15185.jpnprd01.prod.outlook.com/3-bytea_test.py)
download | inline:
from time import time
from sys import argv
import psycopg2
# import psycopg2.extras
"""
Usage
Set the input_path, output_path and conn_params in the main guard below.
To create the test table
python3 bytea_test.py c
To insert a file in the input_path N times
python3 bytea_test.py w <N>
To read the data from the table and write the first file to output_path
python3 bytea_test.py r
"""
def read_binary(input_path):
with open(input_path, 'rb') as f:
return f.read()
def create_table(conn_params):
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
cursor.execute("""CREATE TABLE IF NOT EXISTS BYTEA_TABLE (data BYTEA);""",)
conn.commit()
cursor.close()
conn.close()
def save_file(binary_data, duplicate, conn_params):
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
data = psycopg2.Binary(binary_data)
# rows = [(data,) for _ in range(duplicate)]
# start = time()
# psycopg2.extras.execute_values(cursor, """INSERT INTO bytea_table (data) VALUES %s;""", rows)
# print(time()-start)
start = time()
for _ in range(duplicate):
cursor.execute("""INSERT INTO bytea_table (data) VALUES (%s);""", (data,))
print(time()-start)
conn.commit()
cursor.close()
conn.close()
print(f"file saved {duplicate} times")
def retrieve_file(output_path, conn_params):
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
start = time()
cursor.execute("""SELECT data FROM bytea_table;""")
print(time()-start)
result = cursor.fetchone()
if result:
binary_data = result[0]
with open(output_path, 'wb') as f:
f.write(binary_data)
print(f"File retrieved, saved to '{output_path}'")
else:
print(f"ERROR")
cursor.close()
conn.close()
if __name__ == '__main__':
input_path = 'INPUT/PATH/file.jpg'
output_path = 'OUTPUT/PATCH/retrieved.jpg'
conn_params = {
'dbname': '',
'user': '',
'password': '',
'host': '',
'port': 0
}
if len(argv) == 1:
print("Specify Mode")
elif argv[1].lower() == "c":
create_table(conn_params)
elif argv[1].lower() == "r":
retrieve_file(output_path, conn_params)
elif argv[1].lower() == 'w' and int(argv[2]) > 0:
save_file(read_binary(input_path), int(argv[2]), conn_params)
else:
print("Invalid Mode")
[application/octet-stream] v1-0001-hex-coding-regress-test.patch (7.4K, ../../OS9PR01MB15185626CA7C6EAF82C93ABA3974FA@OS9PR01MB15185.jpnprd01.prod.outlook.com/4-v1-0001-hex-coding-regress-test.patch)
download | inline diff:
From 9066e3296160af4e703f90f460f8e75471b6425d Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Sun, 6 Jul 2025 19:25:28 +0530
Subject: [PATCH v1] hex coding regress test
---
src/test/regress/expected/hex_coding.out | 63 ++++++++++++++++++++++++
src/test/regress/parallel_schedule | 5 ++
src/test/regress/sql/hex_coding.sql | 39 +++++++++++++++
3 files changed, 107 insertions(+)
create mode 100644 src/test/regress/expected/hex_coding.out
create mode 100644 src/test/regress/sql/hex_coding.sql
diff --git a/src/test/regress/expected/hex_coding.out b/src/test/regress/expected/hex_coding.out
new file mode 100644
index 00000000000..e6d78fa4876
--- /dev/null
+++ b/src/test/regress/expected/hex_coding.out
@@ -0,0 +1,63 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d');
+ ^
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+ encode
+----------------------------------------------------------------------------------------------------------------------------------
+ ab
+ 01ab
+ deadc0de
+ baadf00d
+ c001c0ffee
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0de
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+(8 rows)
+
+DROP TABLE BYTEA_TABLE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index a424be2a6bf..8812d80d592 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -109,6 +109,11 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson sqljson_queryfuncs sqljson_jsontable
+# ----------
+# Another group of parallel tests for hex encode/decode
+# ----------
+test: hex_coding
+
# ----------
# Another group of parallel tests
# with depends on create_misc
diff --git a/src/test/regress/sql/hex_coding.sql b/src/test/regress/sql/hex_coding.sql
new file mode 100644
index 00000000000..97c51b62e90
--- /dev/null
+++ b/src/test/regress/sql/hex_coding.sql
@@ -0,0 +1,39 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+
+DROP TABLE BYTEA_TABLE;
--
2.34.1
[application/octet-stream] v5-0001-SVE-support-for-hex-coding.patch (24.6K, ../../OS9PR01MB15185626CA7C6EAF82C93ABA3974FA@OS9PR01MB15185.jpnprd01.prod.outlook.com/5-v5-0001-SVE-support-for-hex-coding.patch)
download | inline diff:
From 5a9bc0e99f7ae102c11cc905cd6c4df6016c415d Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Sun, 6 Jul 2025 19:35:46 +0530
Subject: [PATCH v5] SVE support for hex coding
---
config/c-compiler.m4 | 85 ++++++++
configure | 104 +++++++++
configure.ac | 9 +
meson.build | 81 +++++++
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/encode.c | 6 +-
src/backend/utils/adt/encode_aarch64.c | 280 +++++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 51 ++++-
10 files changed, 615 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/adt/encode_aarch64.c
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index da40bd6a647..73d12826698 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -798,3 +798,88 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_SVE_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the SVE intrinsic required for hex coding:
+# svsub_x, svcmplt, svsel, svcmpgt, svtbl, svlsr_x, svand_z, svcreate2,
+# svptest_any, svnot_z, svorr_z, svcntb, svld1, svwhilelt_b8, svst2, svld2,
+# svget2, svst1 and svlsl_x.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+
+ char input@<:@64@:>@;
+ char output@<:@128@:>@;
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index 16ef5b58d1a..df78a5408d3 100755
--- a/configure
+++ b/configure
@@ -17851,6 +17851,110 @@ $as_echo "#define USE_SVE_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_x, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_x, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+
+ char input[64];
+ char output[128];
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_HEX_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index b3efc49c97a..ce0015bb543 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2107,6 +2107,15 @@ if test x"$host_cpu" = x"aarch64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_HEX_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index a97854a947d..68700a4bba0 100644
--- a/meson.build
+++ b/meson.build
@@ -2389,6 +2389,87 @@ int main(void)
endif
+###############################################################
+# Check the availability of SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+
+char input[64];
+char output[128];
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+}
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char hextbl[] = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+}
+'''
+
+ if cc.links(prog, name: 'SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_HEX_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index ffeacf2b819..d2fa03efe98 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -33,6 +33,7 @@ OBJS = \
dbsize.o \
domains.o \
encode.o \
+ encode_aarch64.o \
enum.o \
expandeddatum.o \
expandedrecord.o \
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d1..fa62ce3107d 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -178,7 +178,7 @@ static const int8 hexlookup[128] = {
};
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +208,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
new file mode 100644
index 00000000000..bf8157900f8
--- /dev/null
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -0,0 +1,280 @@
+/*-------------------------------------------------------------------------
+ *
+ * encode_aarch64.c
+ * Holds the SVE hex encode/decode implementations.
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/encode_aarch64.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <c.h>
+
+#include "utils/builtins.h"
+
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+
+/*
+ * These are the SVE implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * The function pointers are initially set to "choose" functions. These
+ * functions will first set the pointers to the right implementations (based on
+ * what the current CPU supports) and then will call the pointer to fulfill the
+ * caller's request.
+ */
+
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext);
+uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext) = hex_decode_safe_choose;
+
+static inline bool
+check_sve_support(void)
+{
+#ifdef HAVE_ELF_AUX_INFO
+ unsigned long value;
+
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL)
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_scalar;
+ hex_decode_optimized = hex_decode_scalar;
+ hex_decode_safe_optimized = hex_decode_safe_scalar;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint32 vec_len = svcntb();
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svbool_t pred = svptrue_b8();
+ size_t loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svuint8_t bytes, high, low;
+ svuint8x2_t zipped;
+
+ for (size_t i = 0; i < loop_bytes; i += 2 * vec_len)
+ {
+ bytes = svld1(pred, (uint8 *) src);
+
+ /* Right-shift to obtain the high nibble */
+ high = svlsr_x(pred, bytes, 4);
+
+ /* Mask the high nibble to obtain the low nibble */
+ low = svand_z(pred, bytes, 0xF);
+
+ /*
+ * Convert the high and low nibbles to hexadecimal digits using a
+ * vectorized table lookup and zip (interleave) the hexadecimal digits.
+ */
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+
+ /* unrolled */
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ /* process remaining tail bytes */
+ for (size_t i = loop_bytes; i < len; i += vec_len)
+ {
+ pred = svwhilelt_b8((uint64) i, (uint64) len);
+ bytes = svld1(pred, (uint8 *) src);
+ high = svlsr_x(pred, bytes, 4);
+ low = svand_z(pred, bytes, 0xF);
+
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+/*
+ * get_hex_sve
+ * Returns true if the hexadecimal digits are successfully converted
+ * to nibbles and stored in 'res'; otherwise, returns false.
+ */
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ /*
+ * Convert ASCII of '0'-'9' to integers 0-9 by subtracting 48 (ASCII of '0').
+ * Similarly, convert letters 'A'–'F' and 'a'–'f' to integers 10–15 by
+ * subtracting 55 ('A' - 10) and 87 ('a' - 10).
+ */
+ svuint8_t digit = svsub_x(pred, vec, '0'),
+ upper = svsub_x(pred, vec, 'A' - 10),
+ lower = svsub_x(pred, vec, 'a' - 10);
+
+ /*
+ * Identify valid values in digits, upper, and lower vectors.
+ * Values 0-9 are valid in digits, while values 10-15 are valid
+ * in upper and lower.
+ *
+ * Example:
+ * vec: '0' '9' 'A' 'F' 'a' 'f'
+ * vec (in ASCII): 48 57 65 70 97 102
+ *
+ * digit: 0 9 17 22 49 54
+ * valid_digit: 1 1 0 0 0 0
+ *
+ * upper: 249 2 10 15 42 47
+ * valid_upper: 0 1 1 1 0 0
+ *
+ * lower: 217 226 234 239 10 15
+ *
+ * Note that values 0-9 are also marked valid in valid_upper, this will be
+ * handled later.
+ */
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+
+ /*
+ * Merge upper and lower vector using the logic: take the element from
+ * upper if it's true in valid_upper else pick the element in lower
+ *
+ * Mark the valid range i.e. 10-15 in letter vector
+ *
+ * letter: 217 2 10 15 10 15
+ * valid_letter: 0 0 1 1 1 1
+ */
+
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+
+ /*
+ * Check for invalid hexadecimal digit. Each value must fall within
+ * the range 0-9 (true in valid_digit) or 10-15 (true in valid_letter) i.e.
+ * the OR of valid_digit and valid_letter should be all true.
+ */
+
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return false;
+
+ /*
+ * Finally, combine digit and letter vectors using the logic:
+ * take the element from digit if it's true in valid_digit else pick the
+ * element in letter.
+ *
+ * res: 0 9 10 15 10 15
+ */
+
+ *res = svsel(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = svcntb();
+ size_t i = 0,
+ loop_bytes = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svbool_t pred = svptrue_b8();
+ const char *p = dst;
+
+ while (i < loop_bytes)
+ {
+ svuint8x2_t bytes = svld2(pred, (uint8 *) src);
+ svuint8_t high = svget2(bytes, 0),
+ low = svget2(bytes, 1);
+
+ /* fallback for characters with ASCII values below '0' */
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+
+ /* fallback if an invalid hexadecimal digit is found */
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ /* form the byte by left-shifting the high nibble and OR-ing it with the low nibble */
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ i += 2 * vec_len;
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > i) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* USE_SVE_HEX_WITH_RUNTIME_CHECK */
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index ed9bbd7b926..094a9c7c013 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -22,6 +22,7 @@ backend_sources += files(
'dbsize.c',
'domains.c',
'encode.c',
+ 'encode_aarch64.c',
'enum.c',
'expandeddatum.c',
'expandedrecord.c',
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 726a7c1be1f..7a227f1875f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -675,6 +675,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_HEX_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d2255..2f72d8df9d1 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,56 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * On AArch64, we can try to use an SVE optimized hex encode/decode on some systems.
+ */
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext);
+#endif
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 16;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-09-03 11:11 ` [email protected] <[email protected]>
2025-09-03 14:48 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: [email protected] @ 2025-09-03 11:11 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Hi all,
Since the CommitFest is underway, could we get some feedback to improve the patch?
_______
Chiranmoy
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-09-03 14:48 ` Nathan Bossart <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Nathan Bossart @ 2025-09-03 14:48 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 03, 2025 at 11:11:24AM +0000, [email protected] wrote:
> Since the CommitFest is underway, could we get some feedback to improve
> the patch?
I see that there was some discussion about a Neon implementation upthread,
but I'm not sure we concluded anything. For popcount, we first added a
Neon version before adding the SVE version, which required more complicated
configure/runtime checks. Presumably Neon is available on more hardware
than SVE, so that could be a good place to start here, too.
Also, I'd strongly encourage you to get involved with others' patches on
the mailing lists (e.g., reviewing, testing). Patch submissions are great,
but this community depends on other types of participation, too. IME
helping others with their patches also tends to incentivize others to help
with yours.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-09-04 06:20 ` John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: John Naylor @ 2025-09-04 06:20 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 3, 2025 at 6:11 PM [email protected]
<[email protected]> wrote:
>
> Hi all,
>
> Since the CommitFest is underway, could we get some feedback to improve the patch?
On that note, I was hoping you could give us feedback on whether the
improvement in PG18 made any difference at all in your real-world
use-case, i.e. not just in a microbenchmark, but also including
transmission of the hex-encoded values across the network to the
client (that I assume must decode them again).
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
@ 2025-09-04 14:55 ` [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-09-04 14:55 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> I see that there was some discussion about a Neon implementation upthread,
> but I'm not sure we concluded anything. For popcount, we first added a
> Neon version before adding the SVE version, which required more complicated
> configure/runtime checks. Presumably Neon is available on more hardware
> than SVE, so that could be a good place to start here, too.
We have added the Neon versions of hex encode/decode.
Here are the microbenchmark numbers.
hex_encode - m7g.4xlarge
Input | Head | Neon
-------+--------+--------
32 | 18.056 | 5.957
40 | 22.127 | 10.205
48 | 26.214 | 14.151
64 | 33.613 | 6.164
128 | 66.060 | 11.372
256 |130.225 | 18.543
512 |267.105 | 33.977
1024 |515.603 | 64.462
hex_decode - m7g.4xlarge
Input | Head | Neon
-------+--------+--------
32 | 26.669 | 9.462
40 | 36.320 | 19.347
48 | 45.971 | 19.099
64 | 58.468 | 17.648
128 |113.250 | 30.437
256 |218.743 | 56.824
512 |414.133 |107.212
1024 |828.493 |210.740
> Also, I'd strongly encourage you to get involved with others' patches on
> the mailing lists (e.g., reviewing, testing). Patch submissions are great,
> but this community depends on other types of participation, too. IME
> helping others with their patches also tends to incentivize others to help
> with yours.
Sure, we will try to test/review patches on areas we have experience.
> On that note, I was hoping you could give us feedback on whether the
> improvement in PG18 made any difference at all in your real-world
> use-case, i.e. not just in a microbenchmark, but also including
> transmission of the hex-encoded values across the network to the
> client (that I assume must decode them again).
Yes, the improvement in v18 did help, check the attached perf graphs.
We used a python script to send and receive binary data from postgres.
For simple select queries on a bytea column, hex_encode was taking
42% of the query execution time in v17, this was reduced to 33% in v18,
resulting in around 18% improvement in overall query time.
The proposed patch further reduces the hex_encode function usage to
5.6%, another 25% improvement in total query time.
We observed similar improvements for insert queries on the bytea column.
hex_decode usage decreased from 15.5% to 5.5%, a 5-8% query level
improvement depending on which storage type is used.
------
Chiranmoy
Attachments:
[application/octet-stream] v6-0001-NEON-support-for-hex-coding.patch (10.2K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/3-v6-0001-NEON-support-for-hex-coding.patch)
download | inline diff:
From e642b6d32d4715c988b6b93d57385a7c0779182d Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Thu, 4 Sep 2025 12:23:24 +0530
Subject: [PATCH v6 1/3] NEON support for hex coding
---
src/backend/utils/adt/Makefile | 1 +
src/backend/utils/adt/encode.c | 6 +-
src/backend/utils/adt/encode_aarch64.c | 195 +++++++++++++++++++++++++
src/backend/utils/adt/meson.build | 1 +
src/include/utils/builtins.h | 57 +++++++-
5 files changed, 254 insertions(+), 6 deletions(-)
create mode 100644 src/backend/utils/adt/encode_aarch64.c
diff --git a/src/backend/utils/adt/Makefile b/src/backend/utils/adt/Makefile
index cc68ac545a5..40eaee14899 100644
--- a/src/backend/utils/adt/Makefile
+++ b/src/backend/utils/adt/Makefile
@@ -33,6 +33,7 @@ OBJS = \
dbsize.o \
domains.o \
encode.o \
+ encode_aarch64.o \
enum.o \
expandeddatum.o \
expandedrecord.o \
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 4ccaed815d1..fa62ce3107d 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -178,7 +178,7 @@ static const int8 hexlookup[128] = {
};
uint64
-hex_encode(const char *src, size_t len, char *dst)
+hex_encode_scalar(const char *src, size_t len, char *dst)
{
const char *end = src + len;
@@ -208,13 +208,13 @@ get_hex(const char *cp, char *out)
}
uint64
-hex_decode(const char *src, size_t len, char *dst)
+hex_decode_scalar(const char *src, size_t len, char *dst)
{
return hex_decode_safe(src, len, dst, NULL);
}
uint64
-hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+hex_decode_safe_scalar(const char *src, size_t len, char *dst, Node *escontext)
{
const char *s,
*srcend;
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
new file mode 100644
index 00000000000..7b0412dc255
--- /dev/null
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -0,0 +1,195 @@
+/*-------------------------------------------------------------------------
+ *
+ * encode_aarch64.c
+ * Holds the AArch64 hex encode/decode implementations.
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/backend/utils/adt/encode_aarch64.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <c.h>
+
+#include "utils/builtins.h"
+
+#ifdef HEX_CODING_AARCH64
+
+#include <arm_neon.h>
+
+uint64
+hex_encode_optimized(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint8x16_t hextbl_vec = vld1q_u8((uint8 *) hextbl);
+ uint8x16x2_t zipped;
+ uint32 vec_len = sizeof(uint8x16_t);
+ size_t chunks_len = len & ~(2 * vec_len - 1);
+
+ for (size_t i = 0; i < chunks_len; i += 2 * vec_len)
+ {
+ uint8x16_t bytes = vld1q_u8((uint8 *) src);
+
+ /* Right-shift by 4 to get the high nibble */
+ uint8x16_t high = vshrq_n_u8(bytes, 4);
+
+ /* Mask the high nibble to get the low nibble */
+ uint8x16_t low = vandq_u8(bytes, vdupq_n_u8(0xF));
+
+ /*
+ * Convert the high and low nibbles to hexadecimal digits using a table
+ * lookup, and then zip (interleave) the resulting digits.
+ */
+ zipped.val[0] = vqtbl1q_u8(hextbl_vec, high);
+ zipped.val[1] = vqtbl1q_u8(hextbl_vec, low);
+ vst2q_u8((uint8 *) dst, zipped);
+
+ src += vec_len;
+ dst += 2 * vec_len;
+
+ /* unrolled */
+ bytes = vld1q_u8((uint8 *) src);
+ high = vshrq_n_u8(bytes, 4);
+ low = vandq_u8(bytes, vdupq_n_u8(0xF));
+
+ zipped.val[0] = vqtbl1q_u8(hextbl_vec, high);
+ zipped.val[1] = vqtbl1q_u8(hextbl_vec, low);
+ vst2q_u8((uint8 *) dst, zipped);
+
+ src += vec_len;
+ dst += 2 * vec_len;
+ }
+
+
+
+ if (len > chunks_len)
+ hex_encode_scalar(src, len - chunks_len, dst);
+
+ return (uint64) len * 2;
+}
+
+/*
+ * get_hex_neon
+ * Returns true if the hexadecimal digits are successfully converted
+ * to nibbles and stored in 'res'; otherwise, returns false.
+ */
+static inline bool
+get_hex_neon(uint8x16_t vec, uint8x16_t *res)
+{
+ /*
+ * Convert ASCII of '0'-'9' to integers 0-9 by subtracting 48 (ASCII of '0').
+ * Similarly, convert letters 'A'–'F' and 'a'–'f' to integers 10–15 by
+ * subtracting 55 ('A' - 10) and 87 ('a' - 10).
+ */
+ uint8x16_t digit = vsubq_u8(vec, vdupq_n_u8('0'));
+ uint8x16_t upper = vsubq_u8(vec, vdupq_n_u8('A' - 10));
+ uint8x16_t lower = vsubq_u8(vec, vdupq_n_u8('a' - 10));
+
+ /*
+ * Identify valid values in digit, upper, and lower vectors.
+ * Values 0-9 are valid in digits, while values 10-15 are valid
+ * in upper and lower.
+ *
+ * Example:
+ * vec: '0' '9' 'A' 'F' 'a' 'f'
+ * vec (in ASCII): 48 57 65 70 97 102
+ *
+ * digit: 0 9 17 22 49 54
+ * valid_digit: 1 1 0 0 0 0
+ *
+ * upper: 249 2 10 15 42 47
+ * valid_upper: 0 1 1 1 0 0
+ *
+ * lower: 217 226 234 239 10 15
+ *
+ * Note that values 0-9 are also marked valid in valid_upper, this will be
+ * handled later.
+ */
+
+ uint8x16_t valid_digit = vcltq_u8(digit, vdupq_n_u8(10));
+ uint8x16_t valid_upper = vcltq_u8(upper, vdupq_n_u8(16));
+
+ /*
+ * Merge upper and lower vector using the logic: pick the element from
+ * upper if it's true in valid_upper else pick the element in lower
+ *
+ * Mark the valid range i.e. 10-15 in letter vector
+ *
+ * letter: 217 2 10 15 10 15
+ * valid_letter: 0 0 1 1 1 1
+ */
+
+ uint8x16_t letter = vbslq_u8(valid_upper, upper, lower);
+ uint8x16_t valid_letter = vandq_u8(vcgtq_u8(letter, vdupq_n_u8(9)),
+ vcltq_u8(letter, vdupq_n_u8(16)));
+
+ /*
+ * Check for invalid hexadecimal digit. Each value must fall within
+ * the range 0-9 (true in valid_digit) or 10-15 (true in valid_letter) i.e.
+ * the OR of valid_digit and valid_letter should be all true.
+ */
+ uint8x16_t invalid_mask = vmvnq_u8(vorrq_u8(valid_digit, valid_letter));
+
+ if (vmaxvq_u8(invalid_mask) != 0)
+ return false;
+
+ /*
+ * Finally, combine digit and letter vectors using the logic:
+ * pick the element from digit if it's true in valid_digit else pick the
+ * element in letter.
+ *
+ * res: 0 9 10 15 10 15
+ */
+ *res = vbslq_u8(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_optimized(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_optimized(src, len, dst, NULL);
+}
+
+uint64
+hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = sizeof(uint8x16_t);
+ size_t i = 0;
+ size_t chunks_len = len & ~(2 * vec_len - 1); /* process 2 x vec_len per iteration */
+ uint8x16_t ascii_zero = vdupq_n_u8('0');
+ const char *p = dst;
+
+ while (i < chunks_len)
+ {
+ uint8x16x2_t bytes = vld2q_u8((uint8 *) src);
+ uint8x16_t high = bytes.val[0]; /* hex digits for high nibble */
+ uint8x16_t low = bytes.val[1]; /* hex digits for low nibble */
+
+ /* fallback for characters with ASCII values below '0' */
+ uint8x16_t is_below_zero = vorrq_u8(vcltq_u8(high, ascii_zero),
+ vcltq_u8(low, ascii_zero));
+ if (vmaxvq_u8(is_below_zero) != 0)
+ break;
+
+ /* fallback if an invalid hexadecimal digit is found */
+ if (!get_hex_neon(high, &high) || !get_hex_neon(low, &low))
+ break;
+
+ /* form the byte by left-shifting the high nibble and OR-ing it with the low nibble */
+ vst1q_u8((uint8 *) dst, vorrq_u8(vshlq_n_u8(high, 4), low));
+
+ i += 2 * vec_len;
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > i) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* HEX_CODING_AARCH64 */
diff --git a/src/backend/utils/adt/meson.build b/src/backend/utils/adt/meson.build
index dac372c3bea..8b106d03d33 100644
--- a/src/backend/utils/adt/meson.build
+++ b/src/backend/utils/adt/meson.build
@@ -22,6 +22,7 @@ backend_sources += files(
'dbsize.c',
'domains.c',
'encode.c',
+ 'encode_aarch64.c',
'enum.c',
'expandeddatum.c',
'expandedrecord.c',
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index 1c98c7d2255..a809fad2771 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -35,11 +35,62 @@ extern int errdatatype(Oid datatypeOid);
extern int errdomainconstraint(Oid datatypeOid, const char *conname);
/* encode.c */
-extern uint64 hex_encode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode(const char *src, size_t len, char *dst);
-extern uint64 hex_decode_safe(const char *src, size_t len, char *dst,
+extern uint64 hex_encode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_scalar(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
Node *escontext);
+/*
+ * On AArch64, we can use Neon instructions if the compiler provides access to
+ * them (as indicated by __ARM_NEON). As in simd.h, we assume that all
+ * available 64-bit hardware has Neon support.
+ */
+#if defined(__aarch64__) && defined(__ARM_NEON)
+#define HEX_CODING_AARCH64 1
+#endif
+
+#ifdef HEX_CODING_AARCH64
+extern uint64 hex_encode_optimized(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_optimized(const char *src, size_t len, char *dst);
+extern uint64 hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext);
+#endif
+
+static inline uint64
+hex_encode(const char *src, size_t len, char *dst)
+{
+#ifdef HEX_CODING_AARCH64
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_encode_optimized(src, len, dst);
+#endif
+ return hex_encode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode(const char *src, size_t len, char *dst)
+{
+#ifdef HEX_CODING_AARCH64
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_optimized(src, len, dst);
+#endif
+ return hex_decode_scalar(src, len, dst);
+}
+
+static inline uint64
+hex_decode_safe(const char *src, size_t len, char *dst, Node *escontext)
+{
+#ifdef HEX_CODING_AARCH64
+ int threshold = 32;
+
+ if (len >= threshold)
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+#endif
+ return hex_decode_safe_scalar(src, len, dst, escontext);
+}
+
/* int.c */
extern int2vector *buildint2vector(const int16 *int2s, int n);
--
2.34.1
[application/octet-stream] v6-0002-SVE-support-for-hex-coding.patch (21.1K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/4-v6-0002-SVE-support-for-hex-coding.patch)
download | inline diff:
From 7330548cc5b5ebdbe5c8fa515f1eb2eebfc7f2c3 Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Thu, 4 Sep 2025 15:33:47 +0530
Subject: [PATCH v6 2/3] SVE support for hex coding
---
config/c-compiler.m4 | 85 +++++++++
configure | 104 +++++++++++
configure.ac | 9 +
meson.build | 81 +++++++++
src/backend/utils/adt/encode_aarch64.c | 231 ++++++++++++++++++++++++-
src/include/pg_config.h.in | 3 +
src/include/utils/builtins.h | 10 +-
7 files changed, 520 insertions(+), 3 deletions(-)
diff --git a/config/c-compiler.m4 b/config/c-compiler.m4
index da40bd6a647..73d12826698 100644
--- a/config/c-compiler.m4
+++ b/config/c-compiler.m4
@@ -798,3 +798,88 @@ if test x"$Ac_cachevar" = x"yes"; then
fi
undefine([Ac_cachevar])dnl
])# PGAC_SVE_POPCNT_INTRINSICS
+
+# PGAC_ARM_SVE_HEX_INTRINSICS
+# ------------------------------
+# Check if the compiler supports the SVE intrinsic required for hex coding:
+# svsub_x, svcmplt, svsel, svcmpgt, svtbl, svlsr_x, svand_z, svcreate2,
+# svptest_any, svnot_z, svorr_z, svcntb, svld1, svwhilelt_b8, svst2, svld2,
+# svget2, svst1 and svlsl_x.
+#
+# If the intrinsics are supported, sets pgac_arm_sve_hex_intrinsics.
+AC_DEFUN([PGAC_ARM_SVE_HEX_INTRINSICS],
+[define([Ac_cachevar], [AS_TR_SH([pgac_cv_arm_sve_hex_intrinsics])])dnl
+AC_CACHE_CHECK([for svtbl, svlsr_x, svand_z, svcreate2, etc], [Ac_cachevar],
+[AC_LINK_IFELSE([AC_LANG_PROGRAM([#include <arm_sve.h>
+
+ char input@<:@64@:>@;
+ char output@<:@128@:>@;
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output@<:@0@:>@;
+ }],
+ [return hex_coding_test();])],
+ [Ac_cachevar=yes],
+ [Ac_cachevar=no])])
+if test x"$Ac_cachevar" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+undefine([Ac_cachevar])dnl
+])# PGAC_ARM_SVE_HEX_INTRINSICS
diff --git a/configure b/configure
index 39c68161cec..60354107f87 100755
--- a/configure
+++ b/configure
@@ -17735,6 +17735,110 @@ $as_echo "#define USE_SVE_POPCNT_WITH_RUNTIME_CHECK 1" >>confdefs.h
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for svtbl, svlsr_x, svand_z, svcreate2, etc" >&5
+$as_echo_n "checking for svtbl, svlsr_x, svand_z, svcreate2, etc... " >&6; }
+if ${pgac_cv_arm_sve_hex_intrinsics+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+#include <arm_sve.h>
+
+ char input[64];
+ char output[128];
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+ {
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+ }
+
+ #if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+ #endif
+ static int hex_coding_test(void)
+ {
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char *hextbl = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+ }
+int
+main ()
+{
+return hex_coding_test();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ pgac_cv_arm_sve_hex_intrinsics=yes
+else
+ pgac_cv_arm_sve_hex_intrinsics=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $pgac_cv_arm_sve_hex_intrinsics" >&5
+$as_echo "$pgac_cv_arm_sve_hex_intrinsics" >&6; }
+if test x"$pgac_cv_arm_sve_hex_intrinsics" = x"yes"; then
+ pgac_arm_sve_hex_intrinsics=yes
+fi
+
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+
+$as_echo "#define USE_SVE_HEX_WITH_RUNTIME_CHECK 1" >>confdefs.h
+
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for _mm_crc32_u8 and _mm_crc32_u32" >&5
diff --git a/configure.ac b/configure.ac
index 066e3976c0a..6ca57b8c4a7 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2136,6 +2136,15 @@ if test x"$host_cpu" = x"aarch64"; then
fi
fi
+# Check for ARM SVE intrinsics for hex coding
+#
+if test x"$host_cpu" = x"aarch64"; then
+ PGAC_ARM_SVE_HEX_INTRINSICS()
+ if test x"$pgac_arm_sve_hex_intrinsics" = x"yes"; then
+ AC_DEFINE(USE_SVE_HEX_WITH_RUNTIME_CHECK, 1, [Define to 1 to use ARM SVE intrinsic for hex coding.])
+ fi
+fi
+
# Check for Intel SSE 4.2 intrinsics to do CRC calculations.
#
PGAC_SSE42_CRC32_INTRINSICS()
diff --git a/meson.build b/meson.build
index ab8101d67b2..ea5392cfc78 100644
--- a/meson.build
+++ b/meson.build
@@ -2372,6 +2372,87 @@ int main(void)
endif
+###############################################################
+# Check the availability of SVE intrinsics for hex coding.
+###############################################################
+
+if host_cpu == 'aarch64'
+
+ prog = '''
+#include <arm_sve.h>
+
+char input[64];
+char output[128];
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48),
+ upper = svsub_x(pred, vec, 55),
+ lower = svsub_x(pred, vec, 87);
+ svbool_t valid_digit = svcmplt(pred, digit, 10),
+ valid_upper = svcmplt(pred, upper, 16);
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return 0;
+ *res = svsel(valid_digit, digit, letter);
+ return 1;
+}
+
+#if defined(__has_attribute) && __has_attribute (target)
+ __attribute__((target("arch=armv8-a+sve")))
+#endif
+int main(void)
+{
+ int len = 64, vec_len = svcntb(), vec_len_x2 = svcntb() * 2;
+ const char hextbl[] = "0123456789abcdef";
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8_t *) hextbl);
+ char *src = input, *dst = output;
+
+ /* hex encode */
+ for (uint64_t i = 0; i < 64; i += vec_len, dst += 2 * vec_len, src += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i, (uint64_t) len);
+ svuint8_t bytes = svld1(pred, (uint8_t *) src),
+ high = svlsr_x(pred, bytes, 4),
+ low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t merged = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8_t *) dst, merged);
+ }
+
+ /* hex decode */
+ len = 128;
+
+ for (int i; i < len; i += vec_len_x2)
+ {
+ svbool_t pred = svwhilelt_b8((uint64_t) i / 2, (uint64_t) len / 2);
+ svuint8x2_t bytes = svld2(pred, (uint8_t *) src + i);
+ svuint8_t high = svget2(bytes, 0), low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ break;
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ break;
+
+ svst1(pred, (uint8_t *) dst + i / 2, svorr_z(pred, svlsl_x(pred, high, 4), low));
+ }
+
+ /* return computed value, to prevent the above being optimized away */
+ return output[0];
+}
+'''
+
+ if cc.links(prog, name: 'SVE hex coding', args: test_c_args)
+ cdata.set('USE_SVE_HEX_WITH_RUNTIME_CHECK', 1)
+ endif
+
+endif
+
+
###############################################################
# Select CRC-32C implementation.
#
diff --git a/src/backend/utils/adt/encode_aarch64.c b/src/backend/utils/adt/encode_aarch64.c
index 7b0412dc255..4f22fb1d6c7 100644
--- a/src/backend/utils/adt/encode_aarch64.c
+++ b/src/backend/utils/adt/encode_aarch64.c
@@ -20,8 +20,229 @@
#include <arm_neon.h>
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+#include <arm_sve.h>
+
+#if defined(HAVE_ELF_AUX_INFO) || defined(HAVE_GETAUXVAL)
+#include <sys/auxv.h>
+#endif
+
+/*
+ * These are the NEON implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_neon(const char *src, size_t len, char *dst);
+static uint64 hex_decode_neon(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_neon(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * These are the SVE implementations of the hex encode/decode functions.
+ */
+static uint64 hex_encode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_sve(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext);
+
+/*
+ * The function pointers are initially set to "choose" functions. These
+ * functions will first set the pointers to the right implementations (based on
+ * what the current CPU supports) and then will call the pointer to fulfill the
+ * caller's request.
+ */
+
+static uint64 hex_encode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_choose(const char *src, size_t len, char *dst);
+static uint64 hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext);
+uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst) = hex_encode_choose;
+uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst) = hex_decode_choose;
+uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext) = hex_decode_safe_choose;
+
+static inline bool
+check_sve_support(void)
+{
+#ifdef HAVE_ELF_AUX_INFO
+ unsigned long value;
+
+ return elf_aux_info(AT_HWCAP, &value, sizeof(value)) == 0 &&
+ (value & HWCAP_SVE) != 0;
+#elif defined(HAVE_GETAUXVAL)
+ return (getauxval(AT_HWCAP) & HWCAP_SVE) != 0;
+#else
+ return false;
+#endif
+}
+
+static inline void
+choose_hex_functions(void)
+{
+ if (check_sve_support())
+ {
+ hex_encode_optimized = hex_encode_sve;
+ hex_decode_optimized = hex_decode_sve;
+ hex_decode_safe_optimized = hex_decode_safe_sve;
+ }
+ else
+ {
+ hex_encode_optimized = hex_encode_neon;
+ hex_decode_optimized = hex_decode_neon;
+ hex_decode_safe_optimized = hex_decode_safe_neon;
+ }
+}
+
+static uint64
+hex_encode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_encode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_choose(const char *src, size_t len, char *dst)
+{
+ choose_hex_functions();
+ return hex_decode_optimized(src, len, dst);
+}
+
+static uint64
+hex_decode_safe_choose(const char *src, size_t len, char *dst, Node *escontext)
+{
+ choose_hex_functions();
+ return hex_decode_safe_optimized(src, len, dst, escontext);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_encode_sve(const char *src, size_t len, char *dst)
+{
+ const char hextbl[] = "0123456789abcdef";
+ uint32 vec_len = svcntb();
+ svuint8_t hextbl_vec = svld1(svwhilelt_b8(0, 16), (uint8 *) hextbl);
+ svbool_t pred_true = svptrue_b8();
+ size_t chunks_len = len & ~(2 * vec_len - 1); /* process 2 * vec_len bytes per iteration */
+
+ for (size_t i = 0; i < chunks_len; i += 2 * vec_len)
+ {
+ svuint8_t bytes = svld1(pred_true, (uint8 *) src);
+ svuint8_t high = svlsr_x(pred_true, bytes, 4);
+ svuint8_t low = svand_z(pred_true, bytes, 0xF);
+ svuint8x2_t zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred_true, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+
+ /* unrolled */
+ bytes = svld1(pred_true, (uint8 *) src);
+ high = svlsr_x(pred_true, bytes, 4);
+ low = svand_z(pred_true, bytes, 0xF);
+ zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred_true, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ /* process remaining tail bytes */
+ for (size_t i = chunks_len; i < len; i += vec_len)
+ {
+ svbool_t pred = svwhilelt_b8((uint64) i, (uint64) len);
+ svuint8_t bytes = svld1(pred, (uint8 *) src);
+ svuint8_t high = svlsr_x(pred, bytes, 4);
+ svuint8_t low = svand_z(pred, bytes, 0xF);
+ svuint8x2_t zipped = svcreate2(svtbl(hextbl_vec, high), svtbl(hextbl_vec, low));
+ svst2(pred, (uint8 *) dst, zipped);
+
+ dst += 2 * vec_len;
+ src += vec_len;
+ }
+
+ return (uint64) len * 2;
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+static inline bool
+get_hex_sve(svbool_t pred, svuint8_t vec, svuint8_t *res)
+{
+ svuint8_t digit = svsub_x(pred, vec, 48);
+ svuint8_t upper = svsub_x(pred, vec, 55);
+ svuint8_t lower = svsub_x(pred, vec, 87);
+
+ svbool_t valid_digit = svcmplt(pred, digit, 10);
+ svbool_t valid_upper = svcmplt(pred, upper, 16);
+
+ svuint8_t letter = svsel(valid_upper, upper, lower);
+ svbool_t valid_letter = svand_z(pred, svcmpgt(pred, letter, 9),
+ svcmplt(pred, letter, 16));
+
+ if (svptest_any(pred, svnot_z(pred, svorr_z(pred, valid_digit, valid_letter))))
+ return false;
+
+ *res = svsel(valid_digit, digit, letter);
+ return true;
+}
+
+uint64
+hex_decode_sve(const char *src, size_t len, char *dst)
+{
+ return hex_decode_safe_sve(src, len, dst, NULL);
+}
+
+pg_attribute_target("arch=armv8-a+sve")
+uint64
+hex_decode_safe_sve(const char *src, size_t len, char *dst, Node *escontext)
+{
+ uint32 vec_len = svcntb();
+ size_t i = 0;
+ size_t chunks_len = len & ~(2 * vec_len - 1); /* process 2 * vec_len byte chunk each iteration */
+ svbool_t pred = svptrue_b8();
+ const char *p = dst;
+ bool fallback = false;
+
+ while (i < chunks_len)
+ {
+ svuint8x2_t bytes = svld2(pred, (uint8 *) src);
+ svuint8_t high = svget2(bytes, 0);
+ svuint8_t low = svget2(bytes, 1);
+
+ if (svptest_any(pred, svorr_z(pred, svcmplt(pred, high, '0'), svcmplt(pred, low, '0'))))
+ {
+ fallback = true;
+ break;
+ }
+
+ if (!get_hex_sve(pred, high, &high) || !get_hex_sve(pred, low, &low))
+ {
+ fallback = true;
+ break;
+ }
+
+ svst1(pred, (uint8 *) dst, svorr_z(pred, svlsl_x(pred, high, 4), low));
+
+ i += 2 * vec_len;
+ src += 2 * vec_len;
+ dst += vec_len;
+ }
+
+ if (len > i && !fallback) /* can use neon for smaller chunks */
+ return dst - p + hex_decode_safe_neon(src, len - i, dst, escontext);
+
+ if (fallback) /* fallback */
+ return dst - p + hex_decode_safe_scalar(src, len - i, dst, escontext);
+
+ return dst - p;
+}
+
+#endif /* USE_SVE_HEX_WITH_RUNTIME_CHECK */
+
+/*
+ * If the compiler supports SVE, rename the NEON versions because the
+ * optimized versions are now referenced via function pointers.
+ */
+
uint64
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+static hex_encode_neon(const char *src, size_t len, char *dst)
+#else
hex_encode_optimized(const char *src, size_t len, char *dst)
+#endif
{
const char hextbl[] = "0123456789abcdef";
uint8x16_t hextbl_vec = vld1q_u8((uint8 *) hextbl);
@@ -63,8 +284,6 @@ hex_encode_optimized(const char *src, size_t len, char *dst)
dst += 2 * vec_len;
}
-
-
if (len > chunks_len)
hex_encode_scalar(src, len - chunks_len, dst);
@@ -148,13 +367,21 @@ get_hex_neon(uint8x16_t vec, uint8x16_t *res)
}
uint64
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+static hex_decode_neon(const char *src, size_t len, char *dst)
+#else
hex_decode_optimized(const char *src, size_t len, char *dst)
+#endif
{
return hex_decode_safe_optimized(src, len, dst, NULL);
}
uint64
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+static hex_decode_safe_neon(const char *src, size_t len, char *dst, Node *escontext)
+#else
hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext)
+#endif
{
uint32 vec_len = sizeof(uint8x16_t);
size_t i = 0;
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index c4dc5d72bdb..a6735bdd21f 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -678,6 +678,9 @@
/* Define to 1 to use AVX-512 popcount instructions with a runtime check. */
#undef USE_AVX512_POPCNT_WITH_RUNTIME_CHECK
+/* Define to 1 to use SVE instructions for hex coding with a runtime check. */
+#undef USE_SVE_HEX_WITH_RUNTIME_CHECK
+
/* Define to 1 to build with Bonjour support. (--with-bonjour) */
#undef USE_BONJOUR
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index a809fad2771..8a80a9ae51f 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -49,7 +49,15 @@ extern uint64 hex_decode_safe_scalar(const char *src, size_t len, char *dst,
#define HEX_CODING_AARCH64 1
#endif
-#ifdef HEX_CODING_AARCH64
+/*
+ * We can try to use an SVE-optimized hex encode/decode on systems supporting SVE.
+ * For that, we use a function pointer.
+ */
+#ifdef USE_SVE_HEX_WITH_RUNTIME_CHECK
+extern PGDLLIMPORT uint64 (*hex_encode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_optimized) (const char *src, size_t len, char *dst);
+extern PGDLLIMPORT uint64 (*hex_decode_safe_optimized) (const char *src, size_t len, char *dst, Node *escontext);
+#elif HEX_CODING_AARCH64
extern uint64 hex_encode_optimized(const char *src, size_t len, char *dst);
extern uint64 hex_decode_optimized(const char *src, size_t len, char *dst);
extern uint64 hex_decode_safe_optimized(const char *src, size_t len, char *dst, Node *escontext);
--
2.34.1
[application/octet-stream] v6-0003-Regression-tests-for-SIMD-hex-coding.patch (7.4K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/5-v6-0003-Regression-tests-for-SIMD-hex-coding.patch)
download | inline diff:
From 2fdd7f1170253984c1b065ac3a0fc43a31997c05 Mon Sep 17 00:00:00 2001
From: Chiranmoy Bhattacharya <[email protected]>
Date: Thu, 4 Sep 2025 15:44:19 +0530
Subject: [PATCH v6 3/3] Regression tests for SIMD hex coding
---
src/test/regress/expected/hex_coding.out | 63 ++++++++++++++++++++++++
src/test/regress/parallel_schedule | 5 ++
src/test/regress/sql/hex_coding.sql | 39 +++++++++++++++
3 files changed, 107 insertions(+)
create mode 100644 src/test/regress/expected/hex_coding.out
create mode 100644 src/test/regress/sql/hex_coding.sql
diff --git a/src/test/regress/expected/hex_coding.out b/src/test/regress/expected/hex_coding.out
new file mode 100644
index 00000000000..e6d78fa4876
--- /dev/null
+++ b/src/test/regress/expected/hex_coding.out
@@ -0,0 +1,63 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE');
+ ^
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+LINE 1: INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d');
+ ^
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+ERROR: invalid hexadecimal data: odd number of digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+ERROR: invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+ERROR: invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+ERROR: invalid hexadecimal digit: " "
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+ encode
+----------------------------------------------------------------------------------------------------------------------------------
+ ab
+ 01ab
+ deadc0de
+ baadf00d
+ c001c0ffee
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0de
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+ deadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0dedeadc0debaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00dbaadf00d
+(8 rows)
+
+DROP TABLE BYTEA_TABLE;
diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule
index fbffc67ae60..876a3988ed0 100644
--- a/src/test/regress/parallel_schedule
+++ b/src/test/regress/parallel_schedule
@@ -109,6 +109,11 @@ test: select_views portals_p2 foreign_key cluster dependency guc bitmapops combo
# ----------
test: json jsonb json_encoding jsonpath jsonpath_encoding jsonb_jsonpath sqljson sqljson_queryfuncs sqljson_jsontable
+# ----------
+# Another group of parallel tests for hex encode/decode
+# ----------
+test: hex_coding
+
# ----------
# Another group of parallel tests
# with depends on create_misc
diff --git a/src/test/regress/sql/hex_coding.sql b/src/test/regress/sql/hex_coding.sql
new file mode 100644
index 00000000000..97c51b62e90
--- /dev/null
+++ b/src/test/regress/sql/hex_coding.sql
@@ -0,0 +1,39 @@
+--
+-- tests for hex_encode and hex_decode in encode.c
+--
+
+-- Build table for testing
+CREATE TABLE BYTEA_TABLE(data BYTEA);
+
+-- hex_decode is used for inserting into bytea column
+-- Set bytea_output to hex so that hex_encode is used and tested
+SET bytea_output = 'hex';
+
+INSERT INTO BYTEA_TABLE VALUES ('\xAB');
+INSERT INTO BYTEA_TABLE VALUES ('\x01ab');
+INSERT INTO BYTEA_TABLE VALUES ('\xDEADC0DE');
+INSERT INTO BYTEA_TABLE VALUES ('\xbaadf00d');
+INSERT INTO BYTEA_TABLE VALUES ('\x C001 c0ffee '); -- hex string with whitespaces
+
+-- errors checking
+INSERT INTO BYTEA_TABLE VALUES ('\xbadf00d'); -- odd number of hex digits
+INSERT INTO BYTEA_TABLE VALUES ('\xdeadcode'); -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC0FFEE'); -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE VALUES ('\xC00LC*DE'); -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE VALUES ('\xbad f00d'); -- invalid hexadecimal digit: " "
+
+-- long hex strings to test SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || repeat('baadf00d', 8))::bytea;
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || ' ' || repeat('baad f00d', 8))::bytea; -- hex string with whitespaces
+
+-- errors checking for SIMD implementation
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'badf00d' || repeat('DEADC0DE', 4))::bytea; -- odd number of hex digits
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'baadfood'|| repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "o"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 4) || 'C00LC0FFEE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "L"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'C00LC*DE' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: "*"
+INSERT INTO BYTEA_TABLE SELECT ('\x' || repeat('DEADC0DE', 8) || 'bad f00d' || repeat('DEADC0DE', 4))::bytea; -- invalid hexadecimal digit: " "
+
+SELECT encode(data, 'hex') FROM BYTEA_TABLE;
+
+DROP TABLE BYTEA_TABLE;
--
2.34.1
[image/svg+xml] bytea_read_hex_encode_sve.svg (292.8K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/6-bytea_read_hex_encode_sve.svg)
download | view image
[image/svg+xml] bytea_read_hex_encode_v17.svg (287.7K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/7-bytea_read_hex_encode_v17.svg)
download | view image
[image/svg+xml] bytea_read_hex_encode_v18.svg (255.1K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/8-bytea_read_hex_encode_v18.svg)
download | view image
[image/svg+xml] bytea_write_hex_decode_sve.svg (325.4K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/9-bytea_write_hex_decode_sve.svg)
download | view image
[image/svg+xml] bytea_write_hex_decode_v18.svg (280.4K, ../../OS9PR01MB15185B278E343A9BA5F0F6AB19700A@OS9PR01MB15185.jpnprd01.prod.outlook.com/10-bytea_write_hex_decode_v18.svg)
download | view image
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-09-11 03:12 ` Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-11 03:12 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: John Naylor <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 04, 2025 at 02:55:50PM +0000, [email protected] wrote:
>> I see that there was some discussion about a Neon implementation upthread,
>> but I'm not sure we concluded anything. For popcount, we first added a
>> Neon version before adding the SVE version, which required more complicated
>> configure/runtime checks. Presumably Neon is available on more hardware
>> than SVE, so that could be a good place to start here, too.
>
> We have added the Neon versions of hex encode/decode.
Thanks. I noticed that this stuff is simple enough that we can use
port/simd.h (with a few added functions). This is especially nice because
it takes care of x86, too. The performance gains look similar to what you
reported for v6:
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 13 | 6 | 54
64 | 34 | 9 | 74
256 | 93 | 25 | 73
1024 | 281 | 78 | 72
4096 | 1086 | 227 | 79
16384 | 4382 | 927 | 79
65536 | 17455 | 3608 | 79
x86
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 10 | 7 | 30
64 | 29 | 9 | 69
256 | 81 | 21 | 74
1024 | 286 | 66 | 77
4096 | 1106 | 253 | 77
16384 | 4383 | 980 | 78
65536 | 17491 | 3886 | 78
I've only modified hex_encode() for now, but I'm optimistic that we can do
something similar for hex_decode().
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-11 10:43 ` [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-09-11 10:43 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: John Naylor <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> Thanks. I noticed that this stuff is simple enough that we can use
> port/simd.h (with a few added functions). This is especially nice because
> it takes care of x86, too. The performance gains look similar to what you
> reported for v6:
This looks good, much cleaner.
One possible improvement would be to use a vectorized table lookup instead of compare and add. I compared v6 and v7 Neon versions, and v6 is always faster.
I’m not sure if SSE2 has a table lookup similar to Neon.
arm - m7g.4xlarge
buf | v6-Neon| v7-Neon| % diff
-------+--------+--------+--------
64 | 6.16 | 8.57 | 28.07
128 | 11.37 | 15.77 | 27.87
256 | 18.54 | 30.28 | 38.77
512 | 33.98 | 62.15 | 45.33
1024 | 64.46 | 117.55 | 45.16
2048 | 124.28 | 254.86 | 51.24
4096 | 243.47 | 509.23 | 52.19
8192 | 487.34 | 953.81 | 48.91
-----
Chiranmoy
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-09-11 15:24 ` Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-11 15:24 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: John Naylor <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 11, 2025 at 10:43:56AM +0000, [email protected] wrote:
> One possible improvement would be to use a vectorized table lookup
> instead of compare and add. I compared v6 and v7 Neon versions, and v6 is
> always faster. I’m not sure if SSE2 has a table lookup similar to Neon.
I'm not finding a simple way to do that kind of table lookup in SSE2. Part
of the reason v6 performs better is because you've unrolled the loop to
process 2 vector's worth of input data in each iteration. This trades
performance with smaller inputs for gains with larger ones. But even if I
do something similar for v7, v6 still wins most of the time.
My current philosophy with this stuff is to favor simplicity,
maintainability, portability, etc. over extracting the absolute maximum
amount of performance gain, so I think we should proceed with the simd.h
approach. But I'm curious how others feel about this.
v8 is an attempt to fix the casting error on MSVC.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-11 15:32 ` Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Tom Lane @ 2025-09-11 15:32 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Nathan Bossart <[email protected]> writes:
> My current philosophy with this stuff is to favor simplicity,
> maintainability, portability, etc. over extracting the absolute maximum
> amount of performance gain, so I think we should proceed with the simd.h
> approach. But I'm curious how others feel about this.
+1. The maintainability aspect is critical over the long run.
Also, there's a very real danger of optimizing for the specific
hardware and test case you are working with, leading to actually
worse performance with future hardware.
regards, tom lane
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
@ 2025-09-12 18:49 ` [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: [email protected] @ 2025-09-12 18:49 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; Tom Lane <[email protected]>; +Cc: John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
> My current philosophy with this stuff is to favor simplicity,
> maintainability, portability, etc. over extracting the absolute maximum
> amount of performance gain, so I think we should proceed with the simd.h
> approach. But I'm curious how others feel about this.
> +1. The maintainability aspect is critical over the long run.
> Also, there's a very real danger of optimizing for the specific
> hardware and test case you are working with, leading to actually
> worse performance with future hardware.
Using simd.h does make it easier to maintain.
Is there a plan to upgrade simd.h to use SSE4 or SSSE3 in the future?
Since SSE2 is much older, it lacks some of the more specialized intrinsics.
For example, vectorized table lookup can be implemented via [0], and
it’s available in SSSE3 and later x86 instruction sets.
[0] https://www.felixcloutier.com/x86/pshufb
-----
Chiranmoy
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-09-12 21:30 ` Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-12 21:30 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Sep 12, 2025 at 06:49:01PM +0000, [email protected] wrote:
> Using simd.h does make it easier to maintain. Is there a plan to upgrade
> simd.h to use SSE4 or SSSE3 in the future? Since SSE2 is much older, it
> lacks some of the more specialized intrinsics. For example, vectorized
> table lookup can be implemented via [0], and it’s available in SSSE3 and
> later x86 instruction sets.
There have been a couple of discussions about the possibility of requiring
x86-64-v2 for Postgres, but I'm not aware of any serious efforts in that
area.
I've attached a new version of the patch with a simd.h version of
hex_decode(). Here are the numbers:
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 22 | 23 | -5
64 | 61 | 23 | 62
256 | 158 | 47 | 70
1024 | 542 | 122 | 77
4096 | 2103 | 429 | 80
16384 | 8548 | 1673 | 80
65536 | 34663 | 6738 | 81
x86
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 13 | 14 | -8
64 | 42 | 15 | 64
256 | 126 | 42 | 67
1024 | 461 | 149 | 68
4096 | 1802 | 576 | 68
16384 | 7166 | 2280 | 68
65536 | 28625 | 9108 | 68
A couple of notes:
* For hex_decode(), we just give up on the SIMD path and fall back on the
scalar path as soon as we see anything outside [0-9A-Za-z]. I suspect
this might introduce a regression for inputs of ~32 to ~64 bytes that
include whitespace (which must be skipped) or invalid characters, but I
don't whether those inputs are common or whether we care.
* The code makes some assumptions about endianness that might not be true
everywhere, but I've yet to dig into this further.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-22 20:05 ` Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-22 20:05 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Sep 12, 2025 at 04:30:21PM -0500, Nathan Bossart wrote:
> I've attached a new version of the patch with a simd.h version of
> hex_decode(). Here are the numbers:
I was able to improve the hex_decode() implementation a bit.
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 11 | 11 | 0
64 | 38 | 7 | 82
256 | 133 | 18 | 86
1024 | 513 | 67 | 87
4096 | 2037 | 271 | 87
16384 | 8326 | 1103 | 87
65536 | 34550 | 4475 | 87
x86
buf | HEAD | patch | % diff
-------+-------+-------+--------
16 | 8 | 9 | -13
64 | 38 | 7 | 82
256 | 121 | 24 | 80
1024 | 457 | 91 | 80
4096 | 1797 | 356 | 80
16384 | 7161 | 1411 | 80
65536 | 28620 | 5632 | 80
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-23 19:02 ` Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-23 19:02 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; John Naylor <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Mon, Sep 22, 2025 at 03:05:44PM -0500, Nathan Bossart wrote:
> I was able to improve the hex_decode() implementation a bit.
I took a closer look at how hex_decode() performs with smaller inputs.
There are some small regressions, so I tried fixing them by adding the
following to the beginning of the function:
if (likely(tail_idx == 0))
return hex_decode_safe_scalar(src, len, dst, escontext);
This helped a little, but it mostly just slowed things down for larger
inputs on AArch64:
arm
buf | HEAD | patch | fix
-------+-------+-------+-------
2 | 4 | 6 | 4
4 | 6 | 7 | 7
8 | 8 | 8 | 8
16 | 11 | 12 | 11
32 | 18 | 5 | 6
64 | 38 | 7 | 8
256 | 134 | 18 | 24
1024 | 514 | 67 | 100
4096 | 2072 | 280 | 389
16384 | 8409 | 1126 | 1537
65536 | 34704 | 4498 | 6128
x86
buf | HEAD | patch | fix
-------+-------+-------+-------
2 | 2 | 2 | 2
4 | 3 | 3 | 3
8 | 4 | 4 | 4
16 | 8 | 9 | 8
32 | 23 | 5 | 5
64 | 37 | 7 | 7
256 | 122 | 24 | 24
1024 | 457 | 91 | 92
4096 | 1798 | 357 | 358
16384 | 7161 | 1411 | 1416
65536 | 28621 | 5630 | 5653
I didn't do this test for hex_encode(), but I'd expect it to follow a
similar pattern. I'm tempted to suggest that these regressions are within
tolerable levels and to forge on with v10. In any case, IMHO this patch is
approaching committable quality, so I'd be grateful for any feedback.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-24 03:59 ` John Naylor <[email protected]>
2025-09-24 21:40 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: John Naylor @ 2025-09-24 03:59 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 24, 2025 at 2:02 AM Nathan Bossart <[email protected]> wrote:
>
> On Mon, Sep 22, 2025 at 03:05:44PM -0500, Nathan Bossart wrote:
> > I was able to improve the hex_decode() implementation a bit.
>
> I took a closer look at how hex_decode() performs with smaller inputs.
> There are some small regressions, so I tried fixing them by adding the
> following to the beginning of the function:
>
> if (likely(tail_idx == 0))
> return hex_decode_safe_scalar(src, len, dst, escontext);
>
> This helped a little, but it mostly just slowed things down for larger
> inputs on AArch64:
> I didn't do this test for hex_encode(), but I'd expect it to follow a
> similar pattern. I'm tempted to suggest that these regressions are within
> tolerable levels and to forge on with v10.
My first thought is, I'd hazard a guess that short byteas are much
less common than short strings.
My second thought is, the decode case is not that critical. From the
end-to-end tests above, the speed of the decode case had a relatively
small global effect compared to the encode case (Perhaps because reads
are cheaper than writes).
+ if (unlikely(!hex_decode_simd_helper(srcv, &dstv1)))
+ break;
But if you really want to do something here, sprinkling "(un)likely"'s
here seems like solving the wrong problem (even if they make any
difference), since the early return is optimizing for exceptional
conditions. In other places (cf. the UTF8 string verifier), we
accumulate errors, and only if we have them at the end do we restart
from the beginning with the slow error-checking path that can show the
user the offending input.
> In any case, IMHO this patch is
> approaching committable quality, so I'd be grateful for any feedback.
+vector8_sssub(const Vector8 v1, const Vector8 v2)
It's hard to parse "sss", so maybe we can borrow an Intel-ism and use
"iss" for the signed case?
+/* vector manipulation */
+#ifndef USE_NO_SIMD
+static inline Vector8 vector8_interleave_low(const Vector8 v1, const
Vector8 v2);
+static inline Vector8 vector8_interleave_high(const Vector8 v1, const
Vector8 v2);
+static inline Vector8 vector8_pack_16(const Vector8 v1, const Vector8 v2);
+static inline Vector32 vector32_shift_left_nibble(const Vector32 v1);
+static inline Vector32 vector32_shift_right_nibble(const Vector32 v1);
+static inline Vector32 vector32_shift_right_byte(const Vector32 v1);
Do we need declarations for these? I recall that the existing
declarations are there for functions that are also used internally.
The nibble/byte things are rather specific. Wouldn't it be more
logical to expose the already-generic shift operations and let the
caller say by how much? Or does the compiler refuse because the
intrinsic doesn't get an immediate value? Some are like that, but I'm
not sure about these. If so, that's annoying and I wonder if there's a
workaround.
+vector8_has_ge(const Vector8 v, const uint8 c)
+{
+#ifdef USE_SSE2
+ Vector8 umax = _mm_max_epu8(v, vector8_broadcast(c));
+ Vector8 cmpe = _mm_cmpeq_epi8(umax, v);
+
+ return vector8_is_highbit_set(cmpe);
We take pains to avoid signed comparison on unsigned input for the
"le" case, and I don't see why it's okay here.
Do the regression tests have long enough cases that test exceptional
paths, like invalid bytes and embedded whitespace? If not, we need
some.
--
John Naylor
Amazon Web Services
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
@ 2025-09-24 21:40 ` Nathan Bossart <[email protected]>
2025-09-25 14:16 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-24 21:40 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Sep 24, 2025 at 10:59:38AM +0700, John Naylor wrote:
> + if (unlikely(!hex_decode_simd_helper(srcv, &dstv1)))
> + break;
>
> But if you really want to do something here, sprinkling "(un)likely"'s
> here seems like solving the wrong problem (even if they make any
> difference), since the early return is optimizing for exceptional
> conditions. In other places (cf. the UTF8 string verifier), we
> accumulate errors, and only if we have them at the end do we restart
> from the beginning with the slow error-checking path that can show the
> user the offending input.
I switched to an accumulator approach in v11.
> +vector8_sssub(const Vector8 v1, const Vector8 v2)
>
> It's hard to parse "sss", so maybe we can borrow an Intel-ism and use
> "iss" for the signed case?
Done.
> +/* vector manipulation */
> +#ifndef USE_NO_SIMD
> +static inline Vector8 vector8_interleave_low(const Vector8 v1, const
> Vector8 v2);
> +static inline Vector8 vector8_interleave_high(const Vector8 v1, const
> Vector8 v2);
> +static inline Vector8 vector8_pack_16(const Vector8 v1, const Vector8 v2);
> +static inline Vector32 vector32_shift_left_nibble(const Vector32 v1);
> +static inline Vector32 vector32_shift_right_nibble(const Vector32 v1);
> +static inline Vector32 vector32_shift_right_byte(const Vector32 v1);
>
> Do we need declarations for these? I recall that the existing
> declarations are there for functions that are also used internally.
Removed.
> The nibble/byte things are rather specific. Wouldn't it be more
> logical to expose the already-generic shift operations and let the
> caller say by how much? Or does the compiler refuse because the
> intrinsic doesn't get an immediate value? Some are like that, but I'm
> not sure about these. If so, that's annoying and I wonder if there's a
> workaround.
Yeah, the compiler refuses unless the value is an integer literal. I
thought of using a switch statement to cover all the values used in-tree,
but I didn't like that, either.
> +vector8_has_ge(const Vector8 v, const uint8 c)
> +{
> +#ifdef USE_SSE2
> + Vector8 umax = _mm_max_epu8(v, vector8_broadcast(c));
> + Vector8 cmpe = _mm_cmpeq_epi8(umax, v);
> +
> + return vector8_is_highbit_set(cmpe);
>
> We take pains to avoid signed comparison on unsigned input for the
> "le" case, and I don't see why it's okay here.
_mm_max_epu8() does unsigned comparisons, I think...
> Do the regression tests have long enough cases that test exceptional
> paths, like invalid bytes and embedded whitespace? If not, we need
> some.
Added.
I've also fixed builds on gcc/arm64, as discussed elsewhere [0]. Here are
the current numbers on my laptop:
arm
buf | HEAD | patch | % diff
-------+-------+-------+--------
2 | 4 | 4 | 0
4 | 6 | 6 | 0
8 | 8 | 8 | 0
16 | 11 | 12 | -9
32 | 18 | 5 | 72
64 | 38 | 6 | 84
256 | 134 | 17 | 87
1024 | 513 | 63 | 88
4096 | 2081 | 262 | 87
16384 | 8524 | 1058 | 88
65536 | 34731 | 4224 | 88
[0] https://postgr.es/m/aNQtN89VW8z-yo3B%40nathan
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-24 21:40 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-25 14:16 ` John Naylor <[email protected]>
2025-09-25 18:50 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: John Naylor @ 2025-09-25 14:16 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 25, 2025 at 4:40 AM Nathan Bossart <[email protected]> wrote:
>
> On Wed, Sep 24, 2025 at 10:59:38AM +0700, John Naylor wrote:
> > + if (unlikely(!hex_decode_simd_helper(srcv, &dstv1)))
> > + break;
> >
> > But if you really want to do something here, sprinkling "(un)likely"'s
> > here seems like solving the wrong problem (even if they make any
> > difference), since the early return is optimizing for exceptional
> > conditions. In other places (cf. the UTF8 string verifier), we
> > accumulate errors, and only if we have them at the end do we restart
> > from the beginning with the slow error-checking path that can show the
> > user the offending input.
>
> I switched to an accumulator approach in v11.
Looks good to me.
+ if (unlikely(!success))
+ i = 0;
This is after the main loop exits, and the cold path is literally one
instruction, so the motivation is not apparent to me.
> > The nibble/byte things are rather specific. Wouldn't it be more
> > logical to expose the already-generic shift operations and let the
> > caller say by how much? Or does the compiler refuse because the
> > intrinsic doesn't get an immediate value? Some are like that, but I'm
> > not sure about these. If so, that's annoying and I wonder if there's a
> > workaround.
>
> Yeah, the compiler refuses unless the value is an integer literal. I
> thought of using a switch statement to cover all the values used in-tree,
> but I didn't like that, either.
Neither option is great, but I mildly lean towards keeping it internal
with "switch" or whatever: By putting the burden of specifying shift
amounts on separately named functions we run a risk of combinatorial
explosion in function names.
If you feel otherwise, I'd at least use actual numbers:
"shift_left_nibble" is an awkward way to say "shift left by 4 bits"
anyway, and also after "byte" and "nibble" there are not many good
English words to convey the operand amount. It's very possible that
needing other shift amounts will never come up, though.
> > +vector8_has_ge(const Vector8 v, const uint8 c)
> > +{
> > +#ifdef USE_SSE2
> > + Vector8 umax = _mm_max_epu8(v, vector8_broadcast(c));
> > + Vector8 cmpe = _mm_cmpeq_epi8(umax, v);
> > +
> > + return vector8_is_highbit_set(cmpe);
> >
> > We take pains to avoid signed comparison on unsigned input for the
> > "le" case, and I don't see why it's okay here.
>
> _mm_max_epu8() does unsigned comparisons, I think...
Ah, I confused myself about what the LE case was avoiding, namely
signed LE, not signed equality on something else.
(Separately, now I'm wondering if we can do the same for
vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
would allow getting rid of )
> > Do the regression tests have long enough cases that test exceptional
> > paths, like invalid bytes and embedded whitespace? If not, we need
> > some.
>
> Added.
Seems comprehensive enough at a glance.
Other comments:
+ * back together to form the final hex-encoded string. It might be
+ * possible to squeeze out a little more gain by manually unrolling the
+ * loop, but for now we don't bother.
My position (and I think the community agrees) is that manual
unrolling is a rare desperation move that has to be justified, so we
don't need to mention its lack.
+ * Some compilers are picky about casts to the same underlying type, and others
+ * are picky about implicit conversions with vector types. This function does
+ * the same thing as vector32_broadcast(), but it returns a Vector8 and is
+ * carefully crafted to avoid compiler indigestion.
+ */
+#ifndef USE_NO_SIMD
+static inline Vector8
+vector8_broadcast_u32(const uint32 c)
+{
+#ifdef USE_SSE2
+ return vector32_broadcast(c);
+#elif defined(USE_NEON)
+ return (Vector8) vector32_broadcast(c);
+#endif
+}
I'm ambivalent about this: The use case doesn't seem well motivated,
since I don't know why we'd actually need to both broadcast arbitrary
integers and also view the result as bytes. Setting arbitrary bytes is
what we're really doing, and would be more likely be useful in the
future (attached, only tested on x86, and I think part of the
strangeness is the endianness you mentioned above). On the other hand,
the Arm workaround results in awful generated code compared to what
you have here. Since the "set" should be hoisted out of the outer
loop, and we already rely on this pattern for vector8_highbit_mask
anyway, it might be tolerable, and we can reduce the pain with bitwise
NOT.
+/*
+ * Pack 16-bit elements in the given vectors into a single vector of 8-bit
+ * elements. NB: The upper 8-bits of each 16-bit element must be zeros, else
+ * this will produce different results on different architectures.
+ */
v10 asserted this requirement -- that still seems like a good thing?
--
John Naylor
Amazon Web Services
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 7ba92c2c481..eb932d4bd8a 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -317,6 +317,9 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ // TODO: set one and use bitwise NOT for the other
+ Vector8 maskupper = vector8_set(0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0);
+ Vector8 masklower = vector8_set(0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff);
Vector8 msk;
bool ret;
@@ -334,9 +337,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0xff00ff00));
+ msk = vector8_and(*dst, masklower);
msk = vector8_shift_right_byte(msk);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x00ff00ff));
+ *dst = vector8_and(*dst, maskupper);
*dst = vector8_shift_left_nibble(*dst);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 531d8b8b6d1..56e810ce081 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,33 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/*
+ * Populate a vector element-wise with the arguments.
+ */
+#ifndef USE_NO_SIMD
+#if defined(USE_NEON)
+// from a patch by Thomas Munro
+static inline Vector8
+vector8_set(uint8 v0, uint8 v1, uint8 v2, uint8 v3,
+ uint8 v4, uint8 v5, uint8 v6, uint8 v7,
+ uint8 v8, uint8 v9, uint8 v10, uint8 v11,
+ uint8 v12, uint8 v13, uint8 v14, uint8 v15)
+{
+ uint8 pg_attribute_aligned(16) values[16] = {
+ v0, v1, v2, v3,
+ v4, v5, v6, v7,
+ v8, v9, v10, v11,
+ v12, v13, v14, v15
+ };
+ return vld1q_u8(values);
+}
+#elif defined(USE_SSE2)
+#ifndef vector8_set
+#define vector8_set(...) _mm_setr_epi8(__VA_ARGS__)
+#endif
+#endif
+#endif /* ! USE_NO_SIMD */
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -368,6 +395,7 @@ vector8_highbit_mask(const Vector8 v)
* returns a uint64, making it inconvenient to combine mask values from
* multiple vectors.
*/
+ // TODO: use vector8_set
static const uint8 mask[16] = {
1 << 0, 1 << 1, 1 << 2, 1 << 3,
1 << 4, 1 << 5, 1 << 6, 1 << 7,
Attachments:
[text/plain] vector8_set.patch.txt (2.4K, ../../CANWCAZb9FhrWf_2OzwXTK_sQeXm0nPz2joimukB-7pVF0SXxXw@mail.gmail.com/2-vector8_set.patch.txt)
download | inline diff:
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 7ba92c2c481..eb932d4bd8a 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -317,6 +317,9 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ // TODO: set one and use bitwise NOT for the other
+ Vector8 maskupper = vector8_set(0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0);
+ Vector8 masklower = vector8_set(0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff, 0, 0xff);
Vector8 msk;
bool ret;
@@ -334,9 +337,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0xff00ff00));
+ msk = vector8_and(*dst, masklower);
msk = vector8_shift_right_byte(msk);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x00ff00ff));
+ *dst = vector8_and(*dst, maskupper);
*dst = vector8_shift_left_nibble(*dst);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 531d8b8b6d1..56e810ce081 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,33 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/*
+ * Populate a vector element-wise with the arguments.
+ */
+#ifndef USE_NO_SIMD
+#if defined(USE_NEON)
+// from a patch by Thomas Munro
+static inline Vector8
+vector8_set(uint8 v0, uint8 v1, uint8 v2, uint8 v3,
+ uint8 v4, uint8 v5, uint8 v6, uint8 v7,
+ uint8 v8, uint8 v9, uint8 v10, uint8 v11,
+ uint8 v12, uint8 v13, uint8 v14, uint8 v15)
+{
+ uint8 pg_attribute_aligned(16) values[16] = {
+ v0, v1, v2, v3,
+ v4, v5, v6, v7,
+ v8, v9, v10, v11,
+ v12, v13, v14, v15
+ };
+ return vld1q_u8(values);
+}
+#elif defined(USE_SSE2)
+#ifndef vector8_set
+#define vector8_set(...) _mm_setr_epi8(__VA_ARGS__)
+#endif
+#endif
+#endif /* ! USE_NO_SIMD */
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -368,6 +395,7 @@ vector8_highbit_mask(const Vector8 v)
* returns a uint64, making it inconvenient to combine mask values from
* multiple vectors.
*/
+ // TODO: use vector8_set
static const uint8 mask[16] = {
1 << 0, 1 << 1, 1 << 2, 1 << 3,
1 << 4, 1 << 5, 1 << 6, 1 << 7,
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-24 21:40 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-25 14:16 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
@ 2025-09-25 18:50 ` Nathan Bossart <[email protected]>
2025-09-29 08:45 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Nathan Bossart @ 2025-09-25 18:50 UTC (permalink / raw)
To: John Naylor <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Thu, Sep 25, 2025 at 09:16:35PM +0700, John Naylor wrote:
> + if (unlikely(!success))
> + i = 0;
>
> This is after the main loop exits, and the cold path is literally one
> instruction, so the motivation is not apparent to me.
Removed. I was thinking about smaller inputs when I added this, but it
probably makes little difference.
>> Yeah, the compiler refuses unless the value is an integer literal. I
>> thought of using a switch statement to cover all the values used in-tree,
>> but I didn't like that, either.
>
> Neither option is great, but I mildly lean towards keeping it internal
> with "switch" or whatever: By putting the burden of specifying shift
> amounts on separately named functions we run a risk of combinatorial
> explosion in function names.
Done.
> (Separately, now I'm wondering if we can do the same for
> vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
> would allow getting rid of )
I think so. I doubt there's any performance advantage, but it could be
nice for code cleanup. (I'm assuming you meant to say vector8_ssub
(renamed to vector8_ussub() in the patch) after "getting rid of.") I'll
do this in the related patch in the "couple of small patches for simd.h"
thread.
> + * back together to form the final hex-encoded string. It might be
> + * possible to squeeze out a little more gain by manually unrolling the
> + * loop, but for now we don't bother.
>
> My position (and I think the community agrees) is that manual
> unrolling is a rare desperation move that has to be justified, so we
> don't need to mention its lack.
Removed.
> + * Some compilers are picky about casts to the same underlying type, and others
> + * are picky about implicit conversions with vector types. This function does
> + * the same thing as vector32_broadcast(), but it returns a Vector8 and is
> + * carefully crafted to avoid compiler indigestion.
> + */
> +#ifndef USE_NO_SIMD
> +static inline Vector8
> +vector8_broadcast_u32(const uint32 c)
> +{
> +#ifdef USE_SSE2
> + return vector32_broadcast(c);
> +#elif defined(USE_NEON)
> + return (Vector8) vector32_broadcast(c);
> +#endif
> +}
>
> I'm ambivalent about this: The use case doesn't seem well motivated,
> since I don't know why we'd actually need to both broadcast arbitrary
> integers and also view the result as bytes. Setting arbitrary bytes is
> what we're really doing, and would be more likely be useful in the
> future (attached, only tested on x86, and I think part of the
> strangeness is the endianness you mentioned above). On the other hand,
> the Arm workaround results in awful generated code compared to what
> you have here. Since the "set" should be hoisted out of the outer
> loop, and we already rely on this pattern for vector8_highbit_mask
> anyway, it might be tolerable, and we can reduce the pain with bitwise
> NOT.
I think I disagree on this one. We're not broadcasting arbitrary bytes for
every vector element, we're broadcasting a patten of bytes that happens to
be wider than the element size. I would expect this to be a relatively
common use-case. Furthermore, the "set" API is closely tethered to the
vector size, which is fine for SSE2/Neon but may not work down the road
(not to mention the USE_NO_SIMD path). Also, the bitwise NOT approach
won't work because we need to use 0x0f000f00 and 0x000f000f to avoid
angering the assertion in vector8_pack_16(), as mentioned below.
> +/*
> + * Pack 16-bit elements in the given vectors into a single vector of 8-bit
> + * elements. NB: The upper 8-bits of each 16-bit element must be zeros, else
> + * this will produce different results on different architectures.
> + */
>
> v10 asserted this requirement -- that still seems like a good thing?
I had removed that because I worried the accumulator approach would cause
it to fail (it does), but looking again, that's easy enough to work around.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-24 21:40 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-25 14:16 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-25 18:50 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
@ 2025-09-29 08:45 ` John Naylor <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: John Naylor @ 2025-09-29 08:45 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: [email protected] <[email protected]>; Tom Lane <[email protected]>; David Rowley <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Fri, Sep 26, 2025 at 1:50 AM Nathan Bossart <[email protected]> wrote:
>
> On Thu, Sep 25, 2025 at 09:16:35PM +0700, John Naylor wrote:
> > (Separately, now I'm wondering if we can do the same for
> > vector8_has_le since _mm_min_epu8 and vminvq_u8 both exist, and that
> > would allow getting rid of )
>
> I think so. I doubt there's any performance advantage, but it could be
> nice for code cleanup. (I'm assuming you meant to say vector8_ssub
> (renamed to vector8_ussub() in the patch) after "getting rid of.")
Yes right, sorry. And it seems good to do such cleanup first, since it
doesn't make sense to rename something that is about to be deleted.
> I think I disagree on this one. We're not broadcasting arbitrary bytes for
> every vector element, we're broadcasting a patten of bytes that happens to
> be wider than the element size. I would expect this to be a relatively
> common use-case.
That's probably true. I'm still worried that the hack for working
around compiler pickiness (while nice enough in it's current form)
might break at some point and require awareness of compiler versions.
Hmm, for this case, we can sidestep the maintainability questions
entirely by instead using the new interleave functions to build the
masks:
vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f))
vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero())
This generates identical code as v12 on Arm and is not bad on x86.
What do you think of the attached?
While looking around again, it looks like the "msk" variable isn't a
mask like the implies to me. Not sure of a better name because I'm not
sure what it represents aside from a temp variable.
+#elif defined(USE_NEON)
+ switch (i)
+ {
+ case 4:
+ return (Vector8) vshrq_n_u32((Vector32) v1, 4);
+ case 8:
+ return (Vector8) vshrq_n_u32((Vector32) v1, 8);
+ default:
+ pg_unreachable();
+ return vector8_broadcast(0);
+ }
This is just a compiler hint, so if the input is not handled I think
it will return the wrong answer rather than alerting the developer, so
we probabaly want "Assert(false)" here.
Other than that, the pack/unpack functions could use some
documentation about which parameter is low/high.
--
John Naylor
Amazon Web Services
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 87126a003b3..db734481a60 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -315,6 +315,8 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ Vector8 mask_hi = vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f));
+ Vector8 mask_lo = vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero());
Vector8 msk;
bool ret;
@@ -332,9 +334,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0x0f000f00));
+ msk = vector8_and(*dst, mask_hi);
msk = vector8_shift_right(msk, 8);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x000f000f));
+ *dst = vector8_and(*dst, mask_lo);
*dst = vector8_shift_left(*dst, 4);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 0261179e9e7..e316233b7aa 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,17 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/* return a zeroed register */
+static inline Vector8
+vector8_zero()
+{
+#if defined(USE_SSE2)
+ return _mm_setzero_si128();
+#elif defined(USE_NEON)
+ return vmovq_n_u8(0);
+#endif
+}
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -170,24 +181,6 @@ vector32_broadcast(const uint32 c)
}
#endif /* ! USE_NO_SIMD */
-/*
- * Some compilers are picky about casts to the same underlying type, and others
- * are picky about implicit conversions with vector types. This function does
- * the same thing as vector32_broadcast(), but it returns a Vector8 and is
- * carefully crafted to avoid compiler indigestion.
- */
-#ifndef USE_NO_SIMD
-static inline Vector8
-vector8_broadcast_u32(const uint32 c)
-{
-#ifdef USE_SSE2
- return vector32_broadcast(c);
-#elif defined(USE_NEON)
- return (Vector8) vector32_broadcast(c);
-#endif
-}
-#endif /* ! USE_NO_SIMD */
-
/*
* Return true if any elements in the vector are equal to the given scalar.
*/
@@ -577,8 +570,9 @@ vector8_interleave_high(const Vector8 v1, const Vector8 v2)
static inline Vector8
vector8_pack_16(const Vector8 v1, const Vector8 v2)
{
- Vector32 mask PG_USED_FOR_ASSERTS_ONLY = vector32_broadcast(0xff00ff00);
+ Vector8 mask PG_USED_FOR_ASSERTS_ONLY;
+ mask = vector8_interleave_low(vector8_zero(), vector8_broadcast(0xff));
Assert(!vector8_has_ge(vector8_and(v1, mask), 1));
Assert(!vector8_has_ge(vector8_and(v2, mask), 1));
#ifdef USE_SSE2
Attachments:
[text/plain] interleave-mask.patch.txt (2.7K, ../../CANWCAZZMaxfO3CLr1_ViUZMknp_M9iJ0LJbw63E4xMRanGiSqg@mail.gmail.com/2-interleave-mask.patch.txt)
download | inline diff:
diff --git a/src/backend/utils/adt/encode.c b/src/backend/utils/adt/encode.c
index 87126a003b3..db734481a60 100644
--- a/src/backend/utils/adt/encode.c
+++ b/src/backend/utils/adt/encode.c
@@ -315,6 +315,8 @@ static inline bool
hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
{
Vector8 sub;
+ Vector8 mask_hi = vector8_interleave_low(vector8_zero(), vector8_broadcast(0x0f));
+ Vector8 mask_lo = vector8_interleave_low(vector8_broadcast(0x0f), vector8_zero());
Vector8 msk;
bool ret;
@@ -332,9 +334,9 @@ hex_decode_simd_helper(const Vector8 src, Vector8 *dst)
*dst = vector8_issub(src, sub);
ret = !vector8_has_ge(*dst, 0x10);
- msk = vector8_and(*dst, vector8_broadcast_u32(0x0f000f00));
+ msk = vector8_and(*dst, mask_hi);
msk = vector8_shift_right(msk, 8);
- *dst = vector8_and(*dst, vector8_broadcast_u32(0x000f000f));
+ *dst = vector8_and(*dst, mask_lo);
*dst = vector8_shift_left(*dst, 4);
*dst = vector8_or(*dst, msk);
return ret;
diff --git a/src/include/port/simd.h b/src/include/port/simd.h
index 0261179e9e7..e316233b7aa 100644
--- a/src/include/port/simd.h
+++ b/src/include/port/simd.h
@@ -101,6 +101,17 @@ static inline Vector8 vector8_min(const Vector8 v1, const Vector8 v2);
static inline Vector32 vector32_eq(const Vector32 v1, const Vector32 v2);
#endif
+/* return a zeroed register */
+static inline Vector8
+vector8_zero()
+{
+#if defined(USE_SSE2)
+ return _mm_setzero_si128();
+#elif defined(USE_NEON)
+ return vmovq_n_u8(0);
+#endif
+}
+
/*
* Load a chunk of memory into the given vector.
*/
@@ -170,24 +181,6 @@ vector32_broadcast(const uint32 c)
}
#endif /* ! USE_NO_SIMD */
-/*
- * Some compilers are picky about casts to the same underlying type, and others
- * are picky about implicit conversions with vector types. This function does
- * the same thing as vector32_broadcast(), but it returns a Vector8 and is
- * carefully crafted to avoid compiler indigestion.
- */
-#ifndef USE_NO_SIMD
-static inline Vector8
-vector8_broadcast_u32(const uint32 c)
-{
-#ifdef USE_SSE2
- return vector32_broadcast(c);
-#elif defined(USE_NEON)
- return (Vector8) vector32_broadcast(c);
-#endif
-}
-#endif /* ! USE_NO_SIMD */
-
/*
* Return true if any elements in the vector are equal to the given scalar.
*/
@@ -577,8 +570,9 @@ vector8_interleave_high(const Vector8 v1, const Vector8 v2)
static inline Vector8
vector8_pack_16(const Vector8 v1, const Vector8 v2)
{
- Vector32 mask PG_USED_FOR_ASSERTS_ONLY = vector32_broadcast(0xff00ff00);
+ Vector8 mask PG_USED_FOR_ASSERTS_ONLY;
+ mask = vector8_interleave_low(vector8_zero(), vector8_broadcast(0xff));
Assert(!vector8_has_ge(vector8_and(v1, mask), 1));
Assert(!vector8_has_ge(vector8_and(v2, mask), 1));
#ifdef USE_SSE2
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: [PATCH] Hex-coding optimizations using SVE on ARM.
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
@ 2025-01-25 22:07 ` Nathan Bossart <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Nathan Bossart @ 2025-01-25 22:07 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: Tom Lane <[email protected]>; David Rowley <[email protected]>; John Naylor <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
On Wed, Jan 22, 2025 at 10:58:09AM +0000, [email protected] wrote:
>> The functions that test the length before potentially calling a function
>> pointer should probably be inlined (see pg_popcount() in pg_bitutils.h).
>> I wouldn't be surprised if some compilers are inlining this stuff
>> already, but it's probably worth being explicit about it.
>
> Should we implement an inline function in "utils/builtins.h", similar to
> pg_popcount()? Currently, we have not modified the header file, everything
> is statically implemented in encode.c.
Yeah, that's what I'm currently thinking we should do.
--
nathan
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
@ 2025-03-11 10:44 Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-11 10:44 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Feb 25, 2025 at 11:59 AM Ashutosh Bapat
<[email protected]> wrote:
>
> On Tue, Feb 11, 2025 at 5:53 PM Ashutosh Bapat
> <[email protected]> wrote:
> >
> > Hi Michael,
> >
> >
> > On Sun, Feb 9, 2025 at 1:25 PM Michael Paquier <[email protected]> wrote:
> > >
> > > On Fri, Feb 07, 2025 at 07:11:25AM +0900, Michael Paquier wrote:
> > > > Okay, thanks for the feedback. We have been relying on diff -u for
> > > > the parts of the tests touched by 0001 for some time now, so if there
> > > > are no objections I would like to apply 0001 in a couple of days.
> > >
> > > This part has been applied as 169208092f5c.
> >
> > Thanks. PFA rebased patches.
>
> PFA rebased patches.
>
> After rebasing I found another bug and reported it at [1].
This bug has been fixed. But now that it's fixed, it's easy to see
another bug related to materialized view statistics. I have reported
it at [2]. That's the fourth bug identified by this test.
>
> For the time being I have added --no-statistics to the pg_dump command
> when taking a dump for comparison.
>
I have not taken out this option because of materialized view bug.
> [1] https://www.postgresql.org/message-id/[email protected]...
[2] https://www.postgresql.org/message-id/[email protected]...
--
Best Wishes,
Ashutosh Bapat
Attachments:
[text/x-patch] 0002-Filter-COPY-statements-with-differing-colum-20250311.patch (5.7K, ../../CAExHW5uQoyOddBKLBBJpfxXqqok=BTeMvt5OpnM6gw0SroiUUw@mail.gmail.com/2-0002-Filter-COPY-statements-with-differing-colum-20250311.patch)
download | inline diff:
From a140d50245249894a49c39a908163e9bac2fe4bb Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Tue, 11 Feb 2025 16:31:10 +0530
Subject: [PATCH 2/3] Filter COPY statements with differing column order
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 10 +---
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 59 +++++++++++++++------
2 files changed, 45 insertions(+), 24 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index c6b99125d9e..e6d8ac9a757 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -581,9 +581,6 @@ sub test_regression_dump_restore
my $dump_file = "$tempdir/regression_dump.$format";
my $restored_db = 'regression_' . $format;
- # Even though we compare only schema from the original and the restored
- # database (See get_dump_for_comparison() for details.), we dump and
- # restore data as well to catch any errors while doing so.
$src_node->command_ok(
[
'pg_dump', "-F$format", '--no-sync',
@@ -645,13 +642,10 @@ sub get_dump_for_comparison
my $dump_adjusted = "${dumpfile}_adjusted";
- # The order of columns in COPY statements dumped from the original database
- # and that from the restored database differs. These differences are hard to
- # adjust. Hence we compare only schema dumps for now.
$node->command_ok(
[
- 'pg_dump', '-s', '--no-sync', '-d',
- $node->connstr($db), '-f', $dumpfile
+ 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ $dumpfile
],
'dump for comparison succeeded');
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
index e3e152b88fa..e00a00d1b2c 100644
--- a/src/test/perl/PostgreSQL/Test/AdjustDump.pm
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -44,22 +44,36 @@ our @EXPORT = qw(
If we take dump of the regression database left behind after running regression
tests, restore the dump, and take dump of the restored regression database, the
-outputs of both the dumps differ. Some regression tests purposefully create
-some child tables in such a way that their column orders differ from column
-orders of their respective parents. In the restored database, however, their
-column orders are same as that of their respective parents. Thus the column
+outputs of both the dumps differ in the following cases. This routine adjusts
+the given dump so that dump outputs from the original and restored database,
+respectively, match.
+
+Case 1: Some regression tests purposefully create child tables in such a way
+that the order of their inherited columns differ from column orders of their
+respective parents. In the restored database, however, the order of their
+inherited columns are same as that of their respective parents. Thus the column
orders of these child tables in the original database and those in the restored
database differ, causing difference in the dump outputs. See MergeAttributes()
-and dumpTableSchema() for details.
-
-This routine rearranges the column declarations in the relevant
-C<CREATE TABLE... INHERITS> statements in the dump file from original database
-to match those from the restored database. We could instead adjust the
-statements in the dump from the restored database to match those from original
-database or adjust both to a canonical order. But we have chosen to adjust the
-statements in the dump from original database for no particular reason.
-
-Additionally it adjusts blank and new lines to avoid noise.
+and dumpTableSchema() for details. This routine rearranges the column
+declarations in the relevant C<CREATE TABLE... INHERITS> statements in the dump
+file from original database to match those from the restored database. We could,
+instead, adjust the statements in the dump from the restored database to match
+those from original database or adjust both to a canonical order. But we have
+chosen to adjust the statements in the dump from original database for no
+particular reason.
+
+Case 2: When dumping COPY statements the columns are ordered by their attribute
+number by fmtCopyColumnList(). If a column is added to a parent table after a
+child has inherited the parent and the child has its own columns, the attribute
+number of the column changes after restoring the child table. This is because
+when executing the dumped C<CREATE TABLE... INHERITS> statement all the parent
+attributes are created before any child attributes. Thus the order of columns in
+COPY statements dumped from the original and the restored databases,
+respectively, differs. Such tables in regression tests are listed below. It is
+hard to adjust the column order in the COPY statement along with the data. Hence
+we just remove such COPY statements from the dump output.
+
+Additionally the routine adjusts blank and new lines to avoid noise.
Arguments:
@@ -84,8 +98,6 @@ sub adjust_regress_dumpfile
# use Unix newlines
$dump =~ s/\r\n/\n/g;
- # Suppress blank lines, as some places in pg_dump emit more or fewer.
- $dump =~ s/\n\n+/\n/g;
# Adjust the CREATE TABLE ... INHERITS statements.
if ($adjust_child_columns)
@@ -122,6 +134,21 @@ sub adjust_regress_dumpfile
'applied public.test_type_diff2_c2 adjustments');
}
+ # Remove COPY statements with differing column order
+ for my $table (
+ 'public\.b_star', 'public\.c_star',
+ 'public\.cc2', 'public\.d_star',
+ 'public\.e_star', 'public\.f_star',
+ 'public\.renamecolumnanother', 'public\.renamecolumnchild',
+ 'public\.test_type_diff2_c1', 'public\.test_type_diff2_c2',
+ 'public\.test_type_diff_c')
+ {
+ $dump =~ s/^COPY\s$table\s\(.+?^\\\.$//sm;
+ }
+
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
return $dump;
}
--
2.34.1
[text/x-patch] 0001-Test-pg_dump-restore-of-regression-objects-20250311.patch (14.4K, ../../CAExHW5uQoyOddBKLBBJpfxXqqok=BTeMvt5OpnM6gw0SroiUUw@mail.gmail.com/3-0001-Test-pg_dump-restore-of-regression-objects-20250311.patch)
download | inline diff:
From 25c4c7e4ee754dd989d3fd8f015c7355fd9992d6 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 27 Jun 2024 10:03:53 +0530
Subject: [PATCH 1/3] Test pg_dump/restore of regression objects
002_pg_upgrade.pl tests pg_upgrade of the regression database left
behind by regression run. Modify it to test dump and restore of the
regression database as well.
Regression database created by regression run contains almost all the
database objects supported by PostgreSQL in various states. Hence the
new testcase covers dump and restore scenarios not covered by individual
dump/restore cases. Till now 002_pg_upgrade only tested dump/restore
through pg_upgrade which only uses binary mode. Many regression tests
mention that they leave objects behind for dump/restore testing but they
are not tested in a non-binary mode. The new testcase closes that
gap.
Testing dump and restore of regression database makes this test run
longer for a relatively smaller benefit. Hence run it only when
explicitly requested by user by specifying "regress_dump_test" in
PG_TEST_EXTRA.
Note For the reviewers:
The new test has uncovered two bugs so far in one year.
1. Introduced by 14e87ffa5c54. Fixed in fd41ba93e4630921a72ed5127cd0d552a8f3f8fc.
2. Introduced by 0413a556990ba628a3de8a0b58be020fd9a14ed0. Reverted in 74563f6b90216180fc13649725179fc119dddeb5.
Author: Ashutosh Bapat
Reviewed by: Michael Pacquire, Daniel Gustafsson, Tom Lane
Discussion: https://www.postgresql.org/message-id/CAExHW5uF5V=Cjecx3_Z=7xfh4rg2Wf61PT+hfquzjBqouRzQJQ@mail.gmail.com
---
doc/src/sgml/regress.sgml | 12 ++
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 142 +++++++++++++++++++-
src/test/perl/Makefile | 2 +
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 134 ++++++++++++++++++
src/test/perl/meson.build | 1 +
5 files changed, 289 insertions(+), 2 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/AdjustDump.pm
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 0e5e8e8f309..237b974b3ab 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,6 +357,18 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>regress_dump_test</literal></term>
+ <listitem>
+ <para>
+ When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
+ tests dump and restore of regression database left behind by the
+ regression run. Not enabled by default because it is time and resource
+ consuming.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index c00cf68d660..c6b99125d9e 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -12,6 +12,7 @@ use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::AdjustUpgrade;
+use PostgreSQL::Test::AdjustDump;
use Test::More;
# Can be changed to test the other modes.
@@ -35,8 +36,8 @@ sub generate_db
"created database with ASCII characters from $from_char to $to_char");
}
-# Filter the contents of a dump before its use in a content comparison.
-# This returns the path to the filtered dump.
+# Filter the contents of a dump before its use in a content comparison for
+# upgrade testing. This returns the path to the filtered dump.
sub filter_dump
{
my ($is_old, $old_version, $dump_file) = @_;
@@ -261,6 +262,21 @@ else
}
}
is($rc, 0, 'regression tests pass');
+
+ # Test dump/restore of the objects left behind by regression. Ideally it
+ # should be done in a separate TAP test, but doing it here saves us one full
+ # regression run.
+ #
+ # This step takes several extra seconds and some extra disk space, so
+ # requires an opt-in with the PG_TEST_EXTRA environment variable.
+ #
+ # Do this while the old cluster is running before it is shut down by the
+ # upgrade test.
+ if ( $ENV{PG_TEST_EXTRA}
+ && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
+ {
+ test_regression_dump_restore($oldnode, %node_params);
+ }
}
# Initialize a new node for the upgrade.
@@ -527,4 +543,126 @@ my $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
compare_files($dump1_filtered, $dump2_filtered,
'old and new dumps match after pg_upgrade');
+# Test dump and restore of objects left behind by the regression run.
+#
+# It is expected that regression tests, which create `regression` database, are
+# run on `src_node`, which in turn, is left in running state. The dump from
+# `src_node` is restored on a fresh node created using given `node_params`.
+# Plain dumps from both the nodes are compared to make sure that all the dumped
+# objects are restored faithfully.
+sub test_regression_dump_restore
+{
+ my ($src_node, %node_params) = @_;
+ my $dst_node = PostgreSQL::Test::Cluster->new('dst_node');
+
+ # Make sure that the source and destination nodes have the same version and
+ # do not use custom install paths. In both the cases, the dump files may
+ # require additional adjustments unknown to code here. Do not run this test
+ # in such a case to avoid utilizing the time and resources unnecessarily.
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default installation");
+ return;
+ }
+
+ # Dump the original database for comparison later.
+ my $src_dump =
+ get_dump_for_comparison($src_node, 'regression', 'src_dump', 1);
+
+ # Setup destination database cluster
+ $dst_node->init(%node_params);
+ # Stabilize stats for comparison.
+ $dst_node->append_conf('postgresql.conf', 'autovacuum = off');
+ $dst_node->start;
+
+ for my $format ('plain', 'tar', 'directory', 'custom')
+ {
+ my $dump_file = "$tempdir/regression_dump.$format";
+ my $restored_db = 'regression_' . $format;
+
+ # Even though we compare only schema from the original and the restored
+ # database (See get_dump_for_comparison() for details.), we dump and
+ # restore data as well to catch any errors while doing so.
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ # Create a new database for restoring dump from every format so that it
+ # is available for debugging in case the test fails.
+ $dst_node->command_ok([ 'createdb', $restored_db ],
+ "created destination database '$restored_db'");
+
+ # Restore into destination database.
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [
+ 'psql', '-d', $dst_node->connstr($restored_db),
+ '-f', $dump_file
+ ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '-d',
+ $dst_node->connstr($restored_db), $dump_file
+ ];
+ }
+ $dst_node->command_ok(@restore_command,
+ "restored dump taken in $format format on destination instance");
+
+ my $dst_dump =
+ get_dump_for_comparison($dst_node, $restored_db,
+ 'dest_dump.' . $format, 0);
+
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database (using $format format) match"
+ );
+ }
+}
+
+# Dump database `db` from the given `node` in plain format and adjust it for
+# comparing dumps from the original and the restored database.
+#
+# `file_prefix` is used to create unique names for all dump files so that they
+# remain available for debugging in case the test fails.
+#
+# `adjust_child_columns` is passed to adjust_regress_dumpfile() which actually
+# adjusts the dump output.
+#
+# The name of the file containting adjusted dump is returned.
+sub get_dump_for_comparison
+{
+ my ($node, $db, $file_prefix, $adjust_child_columns) = @_;
+
+ my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
+ my $dump_adjusted = "${dumpfile}_adjusted";
+
+
+ # The order of columns in COPY statements dumped from the original database
+ # and that from the restored database differs. These differences are hard to
+ # adjust. Hence we compare only schema dumps for now.
+ $node->command_ok(
+ [
+ 'pg_dump', '-s', '--no-sync', '-d',
+ $node->connstr($db), '-f', $dumpfile
+ ],
+ 'dump for comparison succeeded');
+
+ open(my $dh, '>', $dump_adjusted)
+ || die
+ "could not open $dump_adjusted for writing the adjusted dump: $!";
+ print $dh adjust_regress_dumpfile(slurp_file($dumpfile),
+ $adjust_child_columns);
+ close($dh);
+
+ return $dump_adjusted;
+}
+
done_testing();
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index d82fb67540e..def89650ead 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustUpgrade.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustDump.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
uninstall:
@@ -36,6 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
endif
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..e3e152b88fa
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,134 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustDump - helper module for dump and restore tests
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::AdjustDump;
+
+ # Adjust contents of dump output file so that dump output from original
+ # regression database and that from the restored regression database match
+ $dump = adjust_regress_dumpfile($dump, $adjust_child_columns);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustDump> encapsulates various hacks needed to
+compare the results of dump and restore tests
+
+=cut
+
+package PostgreSQL::Test::AdjustDump;
+
+use strict;
+use warnings FATAL => 'all';
+
+use Exporter 'import';
+use Test::More;
+
+our @EXPORT = qw(
+ adjust_regress_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $dump = adjust_regress_dumpfile($dump, $adjust_child_columns)
+
+If we take dump of the regression database left behind after running regression
+tests, restore the dump, and take dump of the restored regression database, the
+outputs of both the dumps differ. Some regression tests purposefully create
+some child tables in such a way that their column orders differ from column
+orders of their respective parents. In the restored database, however, their
+column orders are same as that of their respective parents. Thus the column
+orders of these child tables in the original database and those in the restored
+database differ, causing difference in the dump outputs. See MergeAttributes()
+and dumpTableSchema() for details.
+
+This routine rearranges the column declarations in the relevant
+C<CREATE TABLE... INHERITS> statements in the dump file from original database
+to match those from the restored database. We could instead adjust the
+statements in the dump from the restored database to match those from original
+database or adjust both to a canonical order. But we have chosen to adjust the
+statements in the dump from original database for no particular reason.
+
+Additionally it adjusts blank and new lines to avoid noise.
+
+Arguments:
+
+=over
+
+=item C<dump>: Contents of dump file
+
+=item C<adjust_child_columns>: 1 indicates that the given dump file requires
+adjusting columns in the child tables; usually when the dump is from original
+database. 0 indicates no such adjustment is needed; usually when the dump is
+from restored database.
+
+=back
+
+Returns the adjusted dump text.
+
+=cut
+
+sub adjust_regress_dumpfile
+{
+ my ($dump, $adjust_child_columns) = @_;
+
+ # use Unix newlines
+ $dump =~ s/\r\n/\n/g;
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
+ # Adjust the CREATE TABLE ... INHERITS statements.
+ if ($adjust_child_columns)
+ {
+ my $saved_dump = $dump;
+
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_stored_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_stored_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_virtual_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_virtual_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c1\s\()
+ (\n\s+int_four\sbigint),
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint)/$1$4,$2,$3/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c1 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c2\s\()
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint),
+ (\n\s+int_four\sbigint)/$1$3,$4,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c2 adjustments');
+ }
+
+ return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff --git a/src/test/perl/meson.build b/src/test/perl/meson.build
index 58e30f15f9d..492ca571ff8 100644
--- a/src/test/perl/meson.build
+++ b/src/test/perl/meson.build
@@ -14,4 +14,5 @@ install_data(
'PostgreSQL/Test/Cluster.pm',
'PostgreSQL/Test/BackgroundPsql.pm',
'PostgreSQL/Test/AdjustUpgrade.pm',
+ 'PostgreSQL/Test/AdjustDump.pm',
install_dir: dir_pgxs / 'src/test/perl/PostgreSQL/Test')
base-commit: dabccf45139a8c7c3c2e7683a943c31077e55a78
--
2.34.1
[text/x-patch] 0003-Do-not-dump-statistics-in-the-file-dumped-f-20250311.patch (1.2K, ../../CAExHW5uQoyOddBKLBBJpfxXqqok=BTeMvt5OpnM6gw0SroiUUw@mail.gmail.com/4-0003-Do-not-dump-statistics-in-the-file-dumped-f-20250311.patch)
download | inline diff:
From 4258ed1bcad537418c4c3f4ba0e3712ec515e09e Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Tue, 25 Feb 2025 11:42:51 +0530
Subject: [PATCH 3/3] Do not dump statistics in the file dumped for comparison
As reported at [1], the dumped and restored statistics may differ if there's a
primary key on the table. Hence do not dump the statistics to avoid differences
in the dump output from the original and restored database.
[1] https://www.postgresql.org/message-id/CAExHW5vf9D+8-a5_BEX3y=2y_xY9hiCxV1=C+FnxDvfprWvkng@mail.gmail.com
Ashutosh Bapat
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index e6d8ac9a757..8924cf8344a 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -644,7 +644,7 @@ sub get_dump_for_comparison
$node->command_ok(
[
- 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ 'pg_dump', '--no-sync', '--no-statistics', '-d', $node->connstr($db), '-f',
$dumpfile
],
'dump for comparison succeeded');
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-12 12:05 ` Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-12 12:05 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hello
When running these tests, I encounter this strange diff in the dumps,
which seems to be that the locale for type money does not match. I
imagine the problem is that the locale is not set correctly when
initdb'ing one of them? Grepping the regress_log for initdb, I see
this:
$ grep -B1 'Running: initdb' tmp_check/log/regress_log_002_pg_upgrade
[13:00:57.580](0.003s) # initializing database system by running initdb
# Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_old_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
--
[13:01:12.879](0.044s) # initializing database system by running initdb
# Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_dst_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
--
[13:01:28.000](0.033s) # initializing database system by running initdb
# Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_new_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding SQL_ASCII --locale-provider libc
[12:50:31.838](0.102s) not ok 15 - dump outputs from original and restored regression database (using plain format) match
[12:50:31.839](0.000s)
[12:50:31.839](0.000s) # Failed test 'dump outputs from original and restored regression database (using plain format) match'
# at /pgsql/source/master/src/test/perl/PostgreSQL/Test/Utils.pm line 797.
[12:50:31.839](0.000s) # got: '1'
# expected: '0'
=== diff of /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/src_dump.sql_adjusted and /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/dest_dump.plain.sql_adjusted
=== stdout ===
--- /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/src_dump.sql_adjusted 2025-03-12 12:50:27.674918597 +0100
+++ /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/dest_dump.plain.sql_adjusted 2025-03-12 12:50:31.778840338 +0100
@@ -208972,7 +208972,7 @@
-- Data for Name: money_data; Type: TABLE DATA; Schema: public; Owner: alvherre
--
COPY public.money_data (m) FROM stdin;
-$123.46
+$ 12.346,00
\.
--
-- Data for Name: mvtest_t; Type: TABLE DATA; Schema: public; Owner: alvherre
@@ -376231,7 +376231,7 @@
-- Data for Name: tab_core_types; Type: TABLE DATA; Schema: public; Owner: alvherre
--
COPY public.tab_core_types (point, line, lseg, box, openedpath, closedpath, polygon, circle, date, "time", "timestamp", timetz, timestamptz, "interval", "json", jsonb, jsonpath, inet, cidr, macaddr8, macaddr, int2, int4, int8, float4, float8, pi, "char", bpchar, "varchar", name, text, bool, bytea, "bit", varbit, money, refcursor, int2vector, oidvector, aclitem, tsvector, tsquery, uuid, xid8, regclass, type, regrole, oid, tid, xid, cid, txid_snapshot, pg_snapshot, pg_lsn, cardinal_number, character_data, sql_identifier, time_stamp, yes_or_no, int4range, int4multirange, int8range, int8multirange, numrange, nummultirange, daterange, datemultirange, tsrange, tsmultirange, tstzrange, tstzmultirange) FROM stdin;
-(11,12) {1,-1,0} [(11,11),(12,12)] (13,13),(11,11) ((11,12),(13,13),(14,14)) [(11,12),(13,13),(14,14)] ((11,12),(13,13),(14,14)) <(1,1),1> 2025-03-12 04:50:14.125899 2025-03-12 04:50:14.125899 04:50:14.125899-07 2025-03-12 12:50:14.125899+01 00:00:12 {"reason":"because"} {"when": "now"} $."a"[*]?(@ > 2) 127.0.0.1 127.0.0.0/8 00:01:03:ff:fe:86:1c:ba 00:01:03:86:1c:ba 2 4 8 4 8 3.14159265358979 f c abc name txt t \\xdeadbeef 1 10001 $12.34 abc 1 2 1 2 alvherre=UC/alvherre 'a' 'and' 'ate' 'cat' 'fat' 'mat' 'on' 'rat' 'sat' 'fat' & 'rat' a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 11 pg_class regtype pg_monitor 1259 (1,1) 2 3 10:20:10,14,15 10:20:10,14,15 16/B374D848 1 l n 2025-03-12 12:50:14.13+01 YES empty {} empty {} (3,4) {(3,4)} [2020-01-03,2021-02-03) {[2020-01-03,2021-02-03)} ("2020-01-02 03:04:05","2021-02-03 06:07:08") {("2020-01-02 03:04:05","2021-02-03 06:07:08")} ("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01") {("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01")}
+(11,12) {1,-1,0} [(11,11),(12,12)] (13,13),(11,11) ((11,12),(13,13),(14,14)) [(11,12),(13,13),(14,14)] ((11,12),(13,13),(14,14)) <(1,1),1> 2025-03-12 04:50:14.125899 2025-03-12 04:50:14.125899 04:50:14.125899-07 2025-03-12 12:50:14.125899+01 00:00:12 {"reason":"because"} {"when": "now"} $."a"[*]?(@ > 2) 127.0.0.1 127.0.0.0/8 00:01:03:ff:fe:86:1c:ba 00:01:03:86:1c:ba 2 4 8 4 8 3.14159265358979 f c abc name txt t \\xdeadbeef 1 10001 $ 1.234,00 abc 1 2 1 2 alvherre=UC/alvherre 'a' 'and' 'ate' 'cat' 'fat' 'mat' 'on' 'rat' 'sat' 'fat' & 'rat' a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 11 pg_class regtype pg_monitor 1259 (1,1) 2 3 10:20:10,14,15 10:20:10,14,15 16/B374D848 1 l n 2025-03-12 12:50:14.13+01 YES empty {} empty {} (3,4) {(3,4)} [2020-01-03,2021-02-03) {[2020-01-03,2021-02-03)} ("2020-01-02 03:04:05","2021-02-03 06:07:08") {("2020-01-02 03:04:05","2021-02-03 06:07:08")} ("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01") {("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01")}
\.
--
-- Data for Name: tableam_parted_a_heap2; Type: TABLE DATA; Schema: public; Owner: alvherre=== stderr ===
=== EOF ===
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"¿Qué importan los años? Lo que realmente importa es comprobar que
a fin de cuentas la mejor edad de la vida es estar vivo" (Mafalda)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-12 15:58 ` Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-12 15:58 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Mar 12, 2025 at 5:35 PM Alvaro Herrera <[email protected]> wrote:
>
> Hello
>
> When running these tests, I encounter this strange diff in the dumps,
> which seems to be that the locale for type money does not match. I
> imagine the problem is that the locale is not set correctly when
> initdb'ing one of them? Grepping the regress_log for initdb, I see
> this:
>
> $ grep -B1 'Running: initdb' tmp_check/log/regress_log_002_pg_upgrade
> [13:00:57.580](0.003s) # initializing database system by running initdb
> # Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_old_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
> --
> [13:01:12.879](0.044s) # initializing database system by running initdb
> # Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_dst_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
> --
> [13:01:28.000](0.033s) # initializing database system by running initdb
> # Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_new_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding SQL_ASCII --locale-provider libc
>
The original node and the node where dump is restored have the same
initdb commands. It's the upgraded node which has different initdb
command. But that's how the test is written originally.
>
>
> [12:50:31.838](0.102s) not ok 15 - dump outputs from original and restored regression database (using plain format) match
> [12:50:31.839](0.000s)
> [12:50:31.839](0.000s) # Failed test 'dump outputs from original and restored regression database (using plain format) match'
> # at /pgsql/source/master/src/test/perl/PostgreSQL/Test/Utils.pm line 797.
> [12:50:31.839](0.000s) # got: '1'
> # expected: '0'
> === diff of /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/src_dump.sql_adjusted and /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/dest_dump.plain.sql_adjusted
> === stdout ===
> --- /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/src_dump.sql_adjusted 2025-03-12 12:50:27.674918597 +0100
> +++ /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/tmp_test_vVew/dest_dump.plain.sql_adjusted 2025-03-12 12:50:31.778840338 +0100
> @@ -208972,7 +208972,7 @@
> -- Data for Name: money_data; Type: TABLE DATA; Schema: public; Owner: alvherre
> --
> COPY public.money_data (m) FROM stdin;
> -$123.46
> +$ 12.346,00
> \.
> --
> -- Data for Name: mvtest_t; Type: TABLE DATA; Schema: public; Owner: alvherre
> @@ -376231,7 +376231,7 @@
> -- Data for Name: tab_core_types; Type: TABLE DATA; Schema: public; Owner: alvherre
> --
> COPY public.tab_core_types (point, line, lseg, box, openedpath, closedpath, polygon, circle, date, "time", "timestamp", timetz, timestamptz, "interval", "json", jsonb, jsonpath, inet, cidr, macaddr8, macaddr, int2, int4, int8, float4, float8, pi, "char", bpchar, "varchar", name, text, bool, bytea, "bit", varbit, money, refcursor, int2vector, oidvector, aclitem, tsvector, tsquery, uuid, xid8, regclass, type, regrole, oid, tid, xid, cid, txid_snapshot, pg_snapshot, pg_lsn, cardinal_number, character_data, sql_identifier, time_stamp, yes_or_no, int4range, int4multirange, int8range, int8multirange, numrange, nummultirange, daterange, datemultirange, tsrange, tsmultirange, tstzrange, tstzmultirange) FROM stdin;
> -(11,12) {1,-1,0} [(11,11),(12,12)] (13,13),(11,11) ((11,12),(13,13),(14,14)) [(11,12),(13,13),(14,14)] ((11,12),(13,13),(14,14)) <(1,1),1> 2025-03-12 04:50:14.125899 2025-03-12 04:50:14.125899 04:50:14.125899-07 2025-03-12 12:50:14.125899+01 00:00:12 {"reason":"because"} {"when": "now"} $."a"[*]?(@ > 2) 127.0.0.1 127.0.0.0/8 00:01:03:ff:fe:86:1c:ba 00:01:03:86:1c:ba 2 4 8 4 8 3.14159265358979 f c abc name txt t \\xdeadbeef 1 10001 $12.34 abc 1 2 1 2 alvherre=UC/alvherre 'a' 'and' 'ate' 'cat' 'fat' 'mat' 'on' 'rat' 'sat' 'fat' & 'rat' a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 11 pg_class regtype pg_monitor 1259 (1,1) 2 3 10:20:10,14,15 10:20:10,14,15 16/B374D848 1 l n 2025-03-12 12:50:14.13+01 YES empty {} empty {} (3,4) {(3,4)} [2020-01-03,2021-02-03) {[2020-01-03,2021-02-03)} ("2020-01-02 03:04:05","2021-02-03 06:07:08") {("2020-01-02 03:04:05","2021-02-03 06:07:08")} ("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01") {("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01")}
> +(11,12) {1,-1,0} [(11,11),(12,12)] (13,13),(11,11) ((11,12),(13,13),(14,14)) [(11,12),(13,13),(14,14)] ((11,12),(13,13),(14,14)) <(1,1),1> 2025-03-12 04:50:14.125899 2025-03-12 04:50:14.125899 04:50:14.125899-07 2025-03-12 12:50:14.125899+01 00:00:12 {"reason":"because"} {"when": "now"} $."a"[*]?(@ > 2) 127.0.0.1 127.0.0.0/8 00:01:03:ff:fe:86:1c:ba 00:01:03:86:1c:ba 2 4 8 4 8 3.14159265358979 f c abc name txt t \\xdeadbeef 1 10001 $ 1.234,00 abc 1 2 1 2 alvherre=UC/alvherre 'a' 'and' 'ate' 'cat' 'fat' 'mat' 'on' 'rat' 'sat' 'fat' & 'rat' a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11 11 pg_class regtype pg_monitor 1259 (1,1) 2 3 10:20:10,14,15 10:20:10,14,15 16/B374D848 1 l n 2025-03-12 12:50:14.13+01 YES empty {} empty {} (3,4) {(3,4)} [2020-01-03,2021-02-03) {[2020-01-03,2021-02-03)} ("2020-01-02 03:04:05","2021-02-03 06:07:08") {("2020-01-02 03:04:05","2021-02-03 06:07:08")} ("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01") {("2020-01-02 12:04:05+01","2021-02-03 15:07:08+01")}
> \.
> --
> -- Data for Name: tableam_parted_a_heap2; Type: TABLE DATA; Schema: public; Owner: alvherre=== stderr ===
> === EOF ===
>
However these differences are coming from original and restored
database which are using the same initdb options.
Does the test pass for you if you don't apply my patches?
Over at [1], I had seen a locale related failure without applying my patches.
[1] https://www.postgresql.org/message-id/[email protected]...
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-12 16:09 ` Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-12 16:09 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-12, Ashutosh Bapat wrote:
> Does the test pass for you if you don't apply my patches?
Yes. It also passes if I keep PG_TEST_EXTRA empty.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-13 06:52 ` Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-13 06:52 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Alvaro,
On Wed, Mar 12, 2025 at 9:39 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-12, Ashutosh Bapat wrote:
>
> > Does the test pass for you if you don't apply my patches?
>
> Yes. It also passes if I keep PG_TEST_EXTRA empty.
I am not able to reproduce this problem locally.
The test uses
In my case the money is printed $<digits before decimal>.<digits after
decimal> format in both the dumps. But in your case the money printed
from restored database has a space between $ and amount and the amount
also has decimal and comma in odd places - I can't figure out what
that means or what lc_monetary value would print something like that.
Can you please help me with
1. can you please run the test again and share the dump outputs. They
will be located in a temporary directory with names
src_dump.sql_adjusted and dest_dump.<format>.sql_adjusted.
2. Are you seeing this diff only with plain format or other formats as well?
Sorry for the trouble.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-13 08:42 ` Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-13 08:42 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hello
On 2025-Mar-13, Ashutosh Bapat wrote:
> 1. can you please run the test again and share the dump outputs. They
> will be located in a temporary directory with names
> src_dump.sql_adjusted and dest_dump.<format>.sql_adjusted.
Ah, I see the problem :-) The first initdb does this:
# Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_old_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
The files belonging to this database system will be owned by user "alvherre".
This user must also own the server process.
The database cluster will be initialized with this locale configuration:
locale provider: builtin
default collation: C.UTF-8
LC_COLLATE: C
LC_CTYPE: C
LC_MESSAGES: C
LC_MONETARY: es_CL.UTF-8
LC_NUMERIC: es_CL.UTF-8
LC_TIME: es_CL.UTF-8
The default text search configuration will be set to "english".
Data page checksums are enabled.
which for some reason used my environment setting for LC_MONETARY.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"But static content is just dynamic content that isn't moving!"
http://smylers.hates-software.com/2007/08/15/fe244d0c.html
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-13 12:39 ` Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-13 12:39 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 13, 2025 at 2:12 PM Alvaro Herrera <[email protected]> wrote:
>
> Hello
>
> On 2025-Mar-13, Ashutosh Bapat wrote:
>
> > 1. can you please run the test again and share the dump outputs. They
> > will be located in a temporary directory with names
> > src_dump.sql_adjusted and dest_dump.<format>.sql_adjusted.
>
> Ah, I see the problem :-) The first initdb does this:
>
> # Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_old_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
> The files belonging to this database system will be owned by user "alvherre".
> This user must also own the server process.
>
> The database cluster will be initialized with this locale configuration:
> locale provider: builtin
> default collation: C.UTF-8
> LC_COLLATE: C
> LC_CTYPE: C
> LC_MESSAGES: C
> LC_MONETARY: es_CL.UTF-8
> LC_NUMERIC: es_CL.UTF-8
> LC_TIME: es_CL.UTF-8
> The default text search configuration will be set to "english".
>
> Data page checksums are enabled.
>
> which for some reason used my environment setting for LC_MONETARY.
>
Thanks. This is super helpful. I am able to reproduce the problem
$ unset LC_MONETARY
$ export PG_TEST_EXTRA=regress_dump_test
$ meson test --suite setup && meson test pg_upgrade/002_pg_upgrade
... snip ...
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
72.38s 44 subtests passed
Ok: 1
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to
/home/ashutosh/work/units/pg_dump_test/build/dev/meson-logs/testlog.txt
$ export LC_MONETARY="es_CL.UTF-8"
$ meson test --suite setup && meson test pg_upgrade/002_pg_upgrade
... snip ...
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade ERROR
69.18s exit status 4
>>> with_icu=no LD_LIBRARY_PATH=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install//home/ashutosh/work/units/pg_dump_test/build/dev/lib/x86_64-linux-gnu REGRESS_SHLIB=/home/ashutosh/work/units/pg_dump_test/build/dev/src/test/regress/regress.so PATH=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install//home/ashutosh/work/units/pg_dump_test/build/dev/bin:/home/ashutosh/work/units/pg_dump_test/build/dev/src/bin/pg_upgrade:/home/ashutosh/work/units/pg_dump_test/build/dev/src/bin/pg_upgrade/test:/home/ashutosh/work/units/pg_dump_test/build/dev/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin MALLOC_PERTURB_=30 share_contrib_dir=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install//home/ashutosh/work/units/pg_dump_test/build/dev/share/postgresql/contrib PG_REGRESS=/home/ashutosh/work/units/pg_dump_test/build/dev/src/test/regress/pg_regress top_builddir=/home/ashutosh/work/units/pg_dump_test/build/dev INITDB_TEMPLATE=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install/initdb-template /usr/bin/python3 /home/ashutosh/work/units/pg_dump_test/build/dev/../../coderoot/pg/src/tools/testwrap --basedir /home/ashutosh/work/units/pg_dump_test/build/dev --srcdir /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/bin/pg_upgrade --pg-test-extra '' --testgroup pg_upgrade --testname 002_pg_upgrade -- /usr/bin/perl -I /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/test/perl -I /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/bin/pg_upgrade /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/bin/pg_upgrade/t/002_pg_upgrade.pl
Ok: 0
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
I see what's happening. If I set LC_MONETARY environment explicitly,
that's taken by initdb
$ export LC_MONETARY="es_CL.UTF-8";rm -rf $DataDir; $BinDir/initdb -D
$DataDir -A trust -N --wal-segsize 1 --allow-group-access --encoding
UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin
--builtin-locale C.UTF-8 -k
The files belonging to this database system will be owned by user "ashutosh".
This user must also own the server process.
The database cluster will be initialized with this locale configuration:
locale provider: builtin
default collation: C.UTF-8
LC_COLLATE: C
LC_CTYPE: C
LC_MESSAGES: en_US.UTF-8
LC_MONETARY: es_CL.UTF-8
LC_NUMERIC: en_US.UTF-8
LC_TIME: en_US.UTF-8
The default text search configuration will be set to "english".
If I don't set it explicitly, it's taken from default settings
$ unset LC_MONETARY;rm -rf $DataDir; $BinDir/initdb -D $DataDir -A
trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8
--lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale
C.UTF-8 -k
The files belonging to this database system will be owned by user "ashutosh".
This user must also own the server process.
The database cluster will be initialized with this locale configuration:
locale provider: builtin
default collation: C.UTF-8
LC_COLLATE: C
LC_CTYPE: C
LC_MESSAGES: en_US.UTF-8
LC_MONETARY: en_US.UTF-8
LC_NUMERIC: en_US.UTF-8
LC_TIME: en_US.UTF-8
The default text search configuration will be set to "english".
In your case probably your default setting is es_CL.UTF-8 or have set
LC_MONETARY explicitly in your environment.
I think the fix is to explicitly pass --lc-monetary to the old cluster
and the restored cluster. 003 patch in the attached patch set does
that. Please check if it fixes the issue for you.
Additionally we should check that it gets copied to the new cluster as
well. But I haven't figured out how to get those settings yet. This
treatment is similar to how --lc-collate and --lc-ctype are treated. I
am wondering whether we should explicitly pass --lc-messages,
--lc-time and --lc-numeric as well.
2d819a08a1cbc11364e36f816b02e33e8dcc030b introduced buildin locale
provider and added overrides to LC_COLLATE and LC_TYPE. But it did not
override other LC_, which I think it should have. In pure upgrade
test, the upgraded node inherits the locale settings of the original
cluster, so this wasn't apparent. But with pg_dump testing, the
original and restored databases are independent. Hence I think we have
to override all LC_* settings by explicitly mentioning --lc-* options
to initdb. Please let me know what you think about this?
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-13 12:40 ` Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-13 12:40 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Here are patches missing in the previous email.
On Thu, Mar 13, 2025 at 6:09 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Thu, Mar 13, 2025 at 2:12 PM Alvaro Herrera <[email protected]> wrote:
> >
> > Hello
> >
> > On 2025-Mar-13, Ashutosh Bapat wrote:
> >
> > > 1. can you please run the test again and share the dump outputs. They
> > > will be located in a temporary directory with names
> > > src_dump.sql_adjusted and dest_dump.<format>.sql_adjusted.
> >
> > Ah, I see the problem :-) The first initdb does this:
> >
> > # Running: initdb -D /home/alvherre/Code/pgsql-build/master/src/bin/pg_upgrade/tmp_check/t_002_pg_upgrade_old_node_data/pgdata -A trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale C.UTF-8 -k
> > The files belonging to this database system will be owned by user "alvherre".
> > This user must also own the server process.
> >
> > The database cluster will be initialized with this locale configuration:
> > locale provider: builtin
> > default collation: C.UTF-8
> > LC_COLLATE: C
> > LC_CTYPE: C
> > LC_MESSAGES: C
> > LC_MONETARY: es_CL.UTF-8
> > LC_NUMERIC: es_CL.UTF-8
> > LC_TIME: es_CL.UTF-8
> > The default text search configuration will be set to "english".
> >
> > Data page checksums are enabled.
> >
> > which for some reason used my environment setting for LC_MONETARY.
> >
>
> Thanks. This is super helpful. I am able to reproduce the problem
> $ unset LC_MONETARY
> $ export PG_TEST_EXTRA=regress_dump_test
> $ meson test --suite setup && meson test pg_upgrade/002_pg_upgrade
> ... snip ...
> 1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
> 72.38s 44 subtests passed
>
>
> Ok: 1
> Expected Fail: 0
> Fail: 0
> Unexpected Pass: 0
> Skipped: 0
> Timeout: 0
>
> Full log written to
> /home/ashutosh/work/units/pg_dump_test/build/dev/meson-logs/testlog.txt
> $ export LC_MONETARY="es_CL.UTF-8"
> $ meson test --suite setup && meson test pg_upgrade/002_pg_upgrade
> ... snip ...
> 1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade ERROR
> 69.18s exit status 4
> >>> with_icu=no LD_LIBRARY_PATH=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install//home/ashutosh/work/units/pg_dump_test/build/dev/lib/x86_64-linux-gnu REGRESS_SHLIB=/home/ashutosh/work/units/pg_dump_test/build/dev/src/test/regress/regress.so PATH=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install//home/ashutosh/work/units/pg_dump_test/build/dev/bin:/home/ashutosh/work/units/pg_dump_test/build/dev/src/bin/pg_upgrade:/home/ashutosh/work/units/pg_dump_test/build/dev/src/bin/pg_upgrade/test:/home/ashutosh/work/units/pg_dump_test/build/dev/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin MALLOC_PERTURB_=30 share_contrib_dir=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install//home/ashutosh/work/units/pg_dump_test/build/dev/share/postgresql/contrib PG_REGRESS=/home/ashutosh/work/units/pg_dump_test/build/dev/src/test/regress/pg_regress top_builddir=/home/ashutosh/work/units/pg_dump_test/build/dev INITDB_TEMPLATE=/home/ashutosh/work/units/pg_dump_test/build/dev/tmp_install/initdb-template /usr/bin/python3 /home/ashutosh/work/units/pg_dump_test/build/dev/../../coderoot/pg/src/tools/testwrap --basedir /home/ashutosh/work/units/pg_dump_test/build/dev --srcdir /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/bin/pg_upgrade --pg-test-extra '' --testgroup pg_upgrade --testname 002_pg_upgrade -- /usr/bin/perl -I /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/test/perl -I /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/bin/pg_upgrade /home/ashutosh/work/units/pg_dump_test/coderoot/pg/src/bin/pg_upgrade/t/002_pg_upgrade.pl
>
>
>
> Ok: 0
> Expected Fail: 0
> Fail: 1
> Unexpected Pass: 0
> Skipped: 0
> Timeout: 0
>
> I see what's happening. If I set LC_MONETARY environment explicitly,
> that's taken by initdb
> $ export LC_MONETARY="es_CL.UTF-8";rm -rf $DataDir; $BinDir/initdb -D
> $DataDir -A trust -N --wal-segsize 1 --allow-group-access --encoding
> UTF-8 --lc-collate C --lc-ctype C --locale-provider builtin
> --builtin-locale C.UTF-8 -k
> The files belonging to this database system will be owned by user "ashutosh".
> This user must also own the server process.
>
> The database cluster will be initialized with this locale configuration:
> locale provider: builtin
> default collation: C.UTF-8
> LC_COLLATE: C
> LC_CTYPE: C
> LC_MESSAGES: en_US.UTF-8
> LC_MONETARY: es_CL.UTF-8
> LC_NUMERIC: en_US.UTF-8
> LC_TIME: en_US.UTF-8
> The default text search configuration will be set to "english".
>
> If I don't set it explicitly, it's taken from default settings
> $ unset LC_MONETARY;rm -rf $DataDir; $BinDir/initdb -D $DataDir -A
> trust -N --wal-segsize 1 --allow-group-access --encoding UTF-8
> --lc-collate C --lc-ctype C --locale-provider builtin --builtin-locale
> C.UTF-8 -k
> The files belonging to this database system will be owned by user "ashutosh".
> This user must also own the server process.
>
> The database cluster will be initialized with this locale configuration:
> locale provider: builtin
> default collation: C.UTF-8
> LC_COLLATE: C
> LC_CTYPE: C
> LC_MESSAGES: en_US.UTF-8
> LC_MONETARY: en_US.UTF-8
> LC_NUMERIC: en_US.UTF-8
> LC_TIME: en_US.UTF-8
> The default text search configuration will be set to "english".
>
> In your case probably your default setting is es_CL.UTF-8 or have set
> LC_MONETARY explicitly in your environment.
>
> I think the fix is to explicitly pass --lc-monetary to the old cluster
> and the restored cluster. 003 patch in the attached patch set does
> that. Please check if it fixes the issue for you.
>
> Additionally we should check that it gets copied to the new cluster as
> well. But I haven't figured out how to get those settings yet. This
> treatment is similar to how --lc-collate and --lc-ctype are treated. I
> am wondering whether we should explicitly pass --lc-messages,
> --lc-time and --lc-numeric as well.
>
> 2d819a08a1cbc11364e36f816b02e33e8dcc030b introduced buildin locale
> provider and added overrides to LC_COLLATE and LC_TYPE. But it did not
> override other LC_, which I think it should have. In pure upgrade
> test, the upgraded node inherits the locale settings of the original
> cluster, so this wasn't apparent. But with pg_dump testing, the
> original and restored databases are independent. Hence I think we have
> to override all LC_* settings by explicitly mentioning --lc-* options
> to initdb. Please let me know what you think about this?
>
> --
> Best Wishes,
> Ashutosh Bapat
--
Best Wishes,
Ashutosh Bapat
Attachments:
[text/x-patch] 0001-Test-pg_dump-restore-of-regression-objects-20250313.patch (16.1K, ../../CAExHW5tOhQi2Fyf5My-YK3uzP8QwVJZQDfC3o-vvAxUUG-CNhg@mail.gmail.com/2-0001-Test-pg_dump-restore-of-regression-objects-20250313.patch)
download | inline diff:
From ec6c178ba0a1a20dc989fee94fdc8d53d531e2e4 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 27 Jun 2024 10:03:53 +0530
Subject: [PATCH 1/3] Test pg_dump/restore of regression objects
002_pg_upgrade.pl tests pg_upgrade of the regression database left
behind by regression run. Modify it to test dump and restore of the
regression database as well.
Regression database created by regression run contains almost all the
database objects supported by PostgreSQL in various states. Hence the
new testcase covers dump and restore scenarios not covered by individual
dump/restore cases. Till now 002_pg_upgrade only tested dump/restore
through pg_upgrade which only uses binary mode. Many regression tests
mention that they leave objects behind for dump/restore testing but they
are not tested in a non-binary mode. The new testcase closes that
gap.
Testing dump and restore of regression database makes this test run
longer for a relatively smaller benefit. Hence run it only when
explicitly requested by user by specifying "regress_dump_test" in
PG_TEST_EXTRA.
Note For the reviewers:
The new test has uncovered two bugs so far in one year.
1. Introduced by 14e87ffa5c54. Fixed in fd41ba93e4630921a72ed5127cd0d552a8f3f8fc.
2. Introduced by 0413a556990ba628a3de8a0b58be020fd9a14ed0. Reverted in 74563f6b90216180fc13649725179fc119dddeb5.
Author: Ashutosh Bapat
Reviewed by: Michael Pacquire, Daniel Gustafsson, Tom Lane, Alvaro Herrera
Discussion: https://www.postgresql.org/message-id/CAExHW5uF5V=Cjecx3_Z=7xfh4rg2Wf61PT+hfquzjBqouRzQJQ@mail.gmail.com
---
doc/src/sgml/regress.sgml | 12 ++
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 141 ++++++++++++++++-
src/test/perl/Makefile | 2 +
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 167 ++++++++++++++++++++
src/test/perl/meson.build | 1 +
5 files changed, 321 insertions(+), 2 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/AdjustDump.pm
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 0e5e8e8f309..237b974b3ab 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,6 +357,18 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>regress_dump_test</literal></term>
+ <listitem>
+ <para>
+ When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
+ tests dump and restore of regression database left behind by the
+ regression run. Not enabled by default because it is time and resource
+ consuming.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index c00cf68d660..bd8313cee6f 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -12,6 +12,7 @@ use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::AdjustUpgrade;
+use PostgreSQL::Test::AdjustDump;
use Test::More;
# Can be changed to test the other modes.
@@ -35,8 +36,8 @@ sub generate_db
"created database with ASCII characters from $from_char to $to_char");
}
-# Filter the contents of a dump before its use in a content comparison.
-# This returns the path to the filtered dump.
+# Filter the contents of a dump before its use in a content comparison for
+# upgrade testing. This returns the path to the filtered dump.
sub filter_dump
{
my ($is_old, $old_version, $dump_file) = @_;
@@ -261,6 +262,21 @@ else
}
}
is($rc, 0, 'regression tests pass');
+
+ # Test dump/restore of the objects left behind by regression. Ideally it
+ # should be done in a separate TAP test, but doing it here saves us one full
+ # regression run.
+ #
+ # This step takes several extra seconds and some extra disk space, so
+ # requires an opt-in with the PG_TEST_EXTRA environment variable.
+ #
+ # Do this while the old cluster is running before it is shut down by the
+ # upgrade test.
+ if ( $ENV{PG_TEST_EXTRA}
+ && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
+ {
+ test_regression_dump_restore($oldnode, %node_params);
+ }
}
# Initialize a new node for the upgrade.
@@ -527,4 +543,125 @@ my $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
compare_files($dump1_filtered, $dump2_filtered,
'old and new dumps match after pg_upgrade');
+# Test dump and restore of objects left behind by the regression run.
+#
+# It is expected that regression tests, which create `regression` database, are
+# run on `src_node`, which in turn, is left in running state. The dump from
+# `src_node` is restored on a fresh node created using given `node_params`.
+# Plain dumps from both the nodes are compared to make sure that all the dumped
+# objects are restored faithfully.
+sub test_regression_dump_restore
+{
+ my ($src_node, %node_params) = @_;
+ my $dst_node = PostgreSQL::Test::Cluster->new('dst_node');
+
+ # Make sure that the source and destination nodes have the same version and
+ # do not use custom install paths. In both the cases, the dump files may
+ # require additional adjustments unknown to code here. Do not run this test
+ # in such a case to avoid utilizing the time and resources unnecessarily.
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default installation");
+ return;
+ }
+
+ # Dump the original database for comparison later.
+ my $src_dump =
+ get_dump_for_comparison($src_node, 'regression', 'src_dump', 1);
+
+ # Setup destination database cluster
+ $dst_node->init(%node_params);
+ # Stabilize stats for comparison.
+ $dst_node->append_conf('postgresql.conf', 'autovacuum = off');
+ $dst_node->start;
+
+ for my $format ('plain', 'tar', 'directory', 'custom')
+ {
+ my $dump_file = "$tempdir/regression_dump.$format";
+ my $restored_db = 'regression_' . $format;
+
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ # Create a new database for restoring dump from every format so that it
+ # is available for debugging in case the test fails.
+ $dst_node->command_ok([ 'createdb', $restored_db ],
+ "created destination database '$restored_db'");
+
+ # Restore into destination database.
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [
+ 'psql', '-d', $dst_node->connstr($restored_db),
+ '-f', $dump_file
+ ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '-d',
+ $dst_node->connstr($restored_db), $dump_file
+ ];
+ }
+ $dst_node->command_ok(@restore_command,
+ "restored dump taken in $format format on destination instance");
+
+ my $dst_dump =
+ get_dump_for_comparison($dst_node, $restored_db,
+ 'dest_dump.' . $format, 0);
+
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database (using $format format) match"
+ );
+ }
+}
+
+# Dump database `db` from the given `node` in plain format and adjust it for
+# comparing dumps from the original and the restored database.
+#
+# `file_prefix` is used to create unique names for all dump files so that they
+# remain available for debugging in case the test fails.
+#
+# `adjust_child_columns` is passed to adjust_regress_dumpfile() which actually
+# adjusts the dump output.
+#
+# The name of the file containting adjusted dump is returned.
+sub get_dump_for_comparison
+{
+ my ($node, $db, $file_prefix, $adjust_child_columns) = @_;
+
+ my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
+ my $dump_adjusted = "${dumpfile}_adjusted";
+
+ # Usually we avoid comparing statistics in our tests since it is flaky by
+ # nature. However, if statistics is dumped and restored it is expected to be
+ # restored as it is i.e. the statistics from the original database and that
+ # from the restored database should match. We turn off autovacuum on the
+ # source and the target database to avoid any statistics update during
+ # restore operation. Hence we do not exclude statistics from dump.
+ $node->command_ok(
+ [
+ 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ $dumpfile
+ ],
+ 'dump for comparison succeeded');
+
+ open(my $dh, '>', $dump_adjusted)
+ || die
+ "could not open $dump_adjusted for writing the adjusted dump: $!";
+ print $dh adjust_regress_dumpfile(slurp_file($dumpfile),
+ $adjust_child_columns);
+ close($dh);
+
+ return $dump_adjusted;
+}
+
done_testing();
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index d82fb67540e..def89650ead 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustUpgrade.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustDump.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
uninstall:
@@ -36,6 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
endif
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..74b9a60cf34
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,167 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustDump - helper module for dump and restore tests
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::AdjustDump;
+
+ # Adjust contents of dump output file so that dump output from original
+ # regression database and that from the restored regression database match
+ $dump = adjust_regress_dumpfile($dump, $adjust_child_columns);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustDump> encapsulates various hacks needed to
+compare the results of dump and restore tests
+
+=cut
+
+package PostgreSQL::Test::AdjustDump;
+
+use strict;
+use warnings FATAL => 'all';
+
+use Exporter 'import';
+use Test::More;
+
+our @EXPORT = qw(
+ adjust_regress_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $dump = adjust_regress_dumpfile($dump, $adjust_child_columns)
+
+If we take dump of the regression database left behind after running regression
+tests, restore the dump, and take dump of the restored regression database, the
+outputs of both the dumps differ in the following cases. This routine adjusts
+the given dump so that dump outputs from the original and restored database,
+respectively, match.
+
+Case 1: Some regression tests purposefully create child tables in such a way
+that the order of their inherited columns differ from column orders of their
+respective parents. In the restored database, however, the order of their
+inherited columns are same as that of their respective parents. Thus the column
+orders of these child tables in the original database and those in the restored
+database differ, causing difference in the dump outputs. See MergeAttributes()
+and dumpTableSchema() for details. This routine rearranges the column
+declarations in the relevant C<CREATE TABLE... INHERITS> statements in the dump
+file from original database to match those from the restored database. We could,
+instead, adjust the statements in the dump from the restored database to match
+those from original database or adjust both to a canonical order. But we have
+chosen to adjust the statements in the dump from original database for no
+particular reason.
+
+Case 2: When dumping COPY statements the columns are ordered by their attribute
+number by fmtCopyColumnList(). If a column is added to a parent table after a
+child has inherited the parent and the child has its own columns, the attribute
+number of the column changes after restoring the child table. This is because
+when executing the dumped C<CREATE TABLE... INHERITS> statement all the parent
+attributes are created before any child attributes. Thus the order of columns in
+COPY statements dumped from the original and the restored databases,
+respectively, differs. Such tables in regression tests are listed below. It is
+hard to adjust the column order in the COPY statement along with the data. Hence
+we just remove such COPY statements from the dump output.
+
+Additionally the routine adjusts blank and new lines to avoid noise.
+
+Note: Usually we avoid comparing statistics in our tests since it is flaky by
+nature. However, if statistics is dumped and restored it is expected to be
+restored as it is i.e. the statistics from the original database and that from
+the restored database should match. Hence we do not filter statistics from dump,
+if it's dumped.
+
+Arguments:
+
+=over
+
+=item C<dump>: Contents of dump file
+
+=item C<adjust_child_columns>: 1 indicates that the given dump file requires
+adjusting columns in the child tables; usually when the dump is from original
+database. 0 indicates no such adjustment is needed; usually when the dump is
+from restored database.
+
+=back
+
+Returns the adjusted dump text.
+
+=cut
+
+sub adjust_regress_dumpfile
+{
+ my ($dump, $adjust_child_columns) = @_;
+
+ # use Unix newlines
+ $dump =~ s/\r\n/\n/g;
+
+ # Adjust the CREATE TABLE ... INHERITS statements.
+ if ($adjust_child_columns)
+ {
+ my $saved_dump = $dump;
+
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_stored_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_stored_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_virtual_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_virtual_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c1\s\()
+ (\n\s+int_four\sbigint),
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint)/$1$4,$2,$3/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c1 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c2\s\()
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint),
+ (\n\s+int_four\sbigint)/$1$3,$4,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c2 adjustments');
+ }
+
+ # Remove COPY statements with differing column order
+ for my $table (
+ 'public\.b_star', 'public\.c_star',
+ 'public\.cc2', 'public\.d_star',
+ 'public\.e_star', 'public\.f_star',
+ 'public\.renamecolumnanother', 'public\.renamecolumnchild',
+ 'public\.test_type_diff2_c1', 'public\.test_type_diff2_c2',
+ 'public\.test_type_diff_c')
+ {
+ $dump =~ s/^COPY\s$table\s\(.+?^\\\.$//sm;
+ }
+
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
+ return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff --git a/src/test/perl/meson.build b/src/test/perl/meson.build
index 58e30f15f9d..492ca571ff8 100644
--- a/src/test/perl/meson.build
+++ b/src/test/perl/meson.build
@@ -14,4 +14,5 @@ install_data(
'PostgreSQL/Test/Cluster.pm',
'PostgreSQL/Test/BackgroundPsql.pm',
'PostgreSQL/Test/AdjustUpgrade.pm',
+ 'PostgreSQL/Test/AdjustDump.pm',
install_dir: dir_pgxs / 'src/test/perl/PostgreSQL/Test')
base-commit: 3691edfab97187789b8a1cbb9dce4acf0ecd8f5a
--
2.34.1
[text/x-patch] 0003-set-lc_monetary-explicitly-at-initdb-time-20250313.patch (1.2K, ../../CAExHW5tOhQi2Fyf5My-YK3uzP8QwVJZQDfC3o-vvAxUUG-CNhg@mail.gmail.com/3-0003-set-lc_monetary-explicitly-at-initdb-time-20250313.patch)
download | inline diff:
From 2f41b4371c0f0b8a3535537c4e4f9ddd1118d1ce Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 13 Mar 2025 16:17:57 +0530
Subject: [PATCH 3/3] set lc_monetary explicitly at initdb time
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 65f4c7d4f2b..51ba79c8589 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -134,6 +134,7 @@ my $original_enc_name;
my $original_provider;
my $original_datcollate = "C";
my $original_datctype = "C";
+my $original_datmonetary = "C";
my $original_datlocale;
if ($oldnode->pg_version >= '17devel')
@@ -163,6 +164,7 @@ my @initdb_params = @custom_opts;
push @initdb_params, ('--encoding', $original_enc_name);
push @initdb_params, ('--lc-collate', $original_datcollate);
push @initdb_params, ('--lc-ctype', $original_datctype);
+push @initdb_params, ('--lc-monetary', $original_datmonetary);
# add --locale-provider, if supported
my %provider_name = ('b' => 'builtin', 'i' => 'icu', 'c' => 'libc');
--
2.34.1
[text/x-patch] 0002-Do-not-dump-statistics-in-the-file-dumped-f-20250313.patch (2.1K, ../../CAExHW5tOhQi2Fyf5My-YK3uzP8QwVJZQDfC3o-vvAxUUG-CNhg@mail.gmail.com/4-0002-Do-not-dump-statistics-in-the-file-dumped-f-20250313.patch)
download | inline diff:
From 75f2b869764d9db3ac5c548636ed5c2be8b47b36 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Tue, 25 Feb 2025 11:42:51 +0530
Subject: [PATCH 2/3] Do not dump statistics in the file dumped for comparison
The dumped and restored statistics of a materialized view may differ as
reported in [1]. Hence do not dump the statistics to avoid differences
in the dump output from the original and restored database.
[1] https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Ashutosh Bapat
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index bd8313cee6f..65f4c7d4f2b 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -641,15 +641,15 @@ sub get_dump_for_comparison
my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
my $dump_adjusted = "${dumpfile}_adjusted";
- # Usually we avoid comparing statistics in our tests since it is flaky by
- # nature. However, if statistics is dumped and restored it is expected to be
- # restored as it is i.e. the statistics from the original database and that
- # from the restored database should match. We turn off autovacuum on the
- # source and the target database to avoid any statistics update during
- # restore operation. Hence we do not exclude statistics from dump.
+ # If statistics is dumped and restored it is expected to be restored as it
+ # is i.e. the statistics from the original database and that from the
+ # restored database should match. We turn off autovacuum on the source and
+ # the target database to avoid any statistics update during restore
+ # operation. But as of now, there are cases when statistics is not being
+ # restored faithfully. Hence for now do not dump statistics.
$node->command_ok(
[
- 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ 'pg_dump', '--no-sync', '--no-statistics', '-d', $node->connstr($db), '-f',
$dumpfile
],
'dump for comparison succeeded');
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-19 11:43 ` Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-19 11:43 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 13, 2025 at 6:10 PM Ashutosh Bapat
<[email protected]> wrote:
> >
> > I think the fix is to explicitly pass --lc-monetary to the old cluster
> > and the restored cluster. 003 patch in the attached patch set does
> > that. Please check if it fixes the issue for you.
> >
> > Additionally we should check that it gets copied to the new cluster as
> > well. But I haven't figured out how to get those settings yet. This
> > treatment is similar to how --lc-collate and --lc-ctype are treated. I
> > am wondering whether we should explicitly pass --lc-messages,
> > --lc-time and --lc-numeric as well.
> >
> > 2d819a08a1cbc11364e36f816b02e33e8dcc030b introduced buildin locale
> > provider and added overrides to LC_COLLATE and LC_TYPE. But it did not
> > override other LC_, which I think it should have. In pure upgrade
> > test, the upgraded node inherits the locale settings of the original
> > cluster, so this wasn't apparent. But with pg_dump testing, the
> > original and restored databases are independent. Hence I think we have
> > to override all LC_* settings by explicitly mentioning --lc-* options
> > to initdb. Please let me know what you think about this?
> >
Investigated this further. The problem is that the pg_regress run
creates regression database with specific properties but the restored
database does not have those properties. That led me to a better
solution. Additionally it's local to the new test. Use --create when
dumping and restoring the regression database. This way the database
properties or "configuration variable settings (as pg_dump
documentation calls them) are copied to the restored database as well.
Those properties include LC_MONETARY. Additionally now the test covers
--create option as well.
PFA patches.
--
Best Wishes,
Ashutosh Bapat
Attachments:
[text/x-patch] 0002-Do-not-dump-statistics-in-the-file-dumped-f-20250319.patch (2.1K, ../../CAExHW5tLVXNVSYwWaq9k8DuYNLZGAVqNzkyZTUCUGQ4OtbD3tQ@mail.gmail.com/2-0002-Do-not-dump-statistics-in-the-file-dumped-f-20250319.patch)
download | inline diff:
From 886e241e304a23bb31b5e59f12149741dfff2b14 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Tue, 25 Feb 2025 11:42:51 +0530
Subject: [PATCH 2/2] Do not dump statistics in the file dumped for comparison
The dumped and restored statistics of a materialized view may differ as
reported in [1]. Hence do not dump the statistics to avoid differences
in the dump output from the original and restored database.
[1] https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Ashutosh Bapat
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index d08eea6693f..f931fef2307 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -656,15 +656,15 @@ sub get_dump_for_comparison
my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
my $dump_adjusted = "${dumpfile}_adjusted";
- # Usually we avoid comparing statistics in our tests since it is flaky by
- # nature. However, if statistics is dumped and restored it is expected to be
- # restored as it is i.e. the statistics from the original database and that
- # from the restored database should match. We turn off autovacuum on the
- # source and the target database to avoid any statistics update during
- # restore operation. Hence we do not exclude statistics from dump.
+ # If statistics is dumped and restored it is expected to be restored as it
+ # is i.e. the statistics from the original database and that from the
+ # restored database should match. We turn off autovacuum on the source and
+ # the target database to avoid any statistics update during restore
+ # operation. But as of now, there are cases when statistics is not being
+ # restored faithfully. Hence for now do not dump statistics.
$node->command_ok(
[
- 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ 'pg_dump', '--no-sync', '--no-statistics', '-d', $node->connstr($db), '-f',
$dumpfile
],
'dump for comparison succeeded');
--
2.34.1
[text/x-patch] 0001-Test-pg_dump-restore-of-regression-objects-20250319.patch (16.5K, ../../CAExHW5tLVXNVSYwWaq9k8DuYNLZGAVqNzkyZTUCUGQ4OtbD3tQ@mail.gmail.com/3-0001-Test-pg_dump-restore-of-regression-objects-20250319.patch)
download | inline diff:
From 1723050dadb89f3187fef19c994d8c866ee5a788 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 27 Jun 2024 10:03:53 +0530
Subject: [PATCH 1/2] Test pg_dump/restore of regression objects
002_pg_upgrade.pl tests pg_upgrade of the regression database left
behind by regression run. Modify it to test dump and restore of the
regression database as well.
Regression database created by regression run contains almost all the
database objects supported by PostgreSQL in various states. Hence the
new testcase covers dump and restore scenarios not covered by individual
dump/restore cases. Till now 002_pg_upgrade only tested dump/restore
through pg_upgrade which only uses binary mode. Many regression tests
mention that they leave objects behind for dump/restore testing but they
are not tested in a non-binary mode. The new testcase closes that
gap.
Testing dump and restore of regression database makes this test run
longer for a relatively smaller benefit. Hence run it only when
explicitly requested by user by specifying "regress_dump_test" in
PG_TEST_EXTRA.
Note For the reviewers:
The new test has uncovered many bugs so far in one year.
1. Introduced by 14e87ffa5c54. Fixed in fd41ba93e4630921a72ed5127cd0d552a8f3f8fc.
2. Introduced by 0413a556990ba628a3de8a0b58be020fd9a14ed0. Reverted in 74563f6b90216180fc13649725179fc119dddeb5.
3. Fixed by d611f8b1587b8f30caa7c0da99ae5d28e914d54f
3. Being discussed on hackers at https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Author: Ashutosh Bapat
Reviewed by: Michael Pacquire, Daniel Gustafsson, Tom Lane, Alvaro Herrera
Discussion: https://www.postgresql.org/message-id/CAExHW5uF5V=Cjecx3_Z=7xfh4rg2Wf61PT+hfquzjBqouRzQJQ@mail.gmail.com
---
doc/src/sgml/regress.sgml | 12 ++
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 144 ++++++++++++++++-
src/test/perl/Makefile | 2 +
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 167 ++++++++++++++++++++
src/test/perl/meson.build | 1 +
5 files changed, 324 insertions(+), 2 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/AdjustDump.pm
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 0e5e8e8f309..237b974b3ab 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,6 +357,18 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>regress_dump_test</literal></term>
+ <listitem>
+ <para>
+ When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
+ tests dump and restore of regression database left behind by the
+ regression run. Not enabled by default because it is time and resource
+ consuming.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 00051b85035..d08eea6693f 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -12,6 +12,7 @@ use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::AdjustUpgrade;
+use PostgreSQL::Test::AdjustDump;
use Test::More;
# Can be changed to test the other modes.
@@ -35,8 +36,8 @@ sub generate_db
"created database with ASCII characters from $from_char to $to_char");
}
-# Filter the contents of a dump before its use in a content comparison.
-# This returns the path to the filtered dump.
+# Filter the contents of a dump before its use in a content comparison for
+# upgrade testing. This returns the path to the filtered dump.
sub filter_dump
{
my ($is_old, $old_version, $dump_file) = @_;
@@ -262,6 +263,21 @@ else
}
}
is($rc, 0, 'regression tests pass');
+
+ # Test dump/restore of the objects left behind by regression. Ideally it
+ # should be done in a separate TAP test, but doing it here saves us one full
+ # regression run.
+ #
+ # This step takes several extra seconds and some extra disk space, so
+ # requires an opt-in with the PG_TEST_EXTRA environment variable.
+ #
+ # Do this while the old cluster is running before it is shut down by the
+ # upgrade test.
+ if ( $ENV{PG_TEST_EXTRA}
+ && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
+ {
+ test_regression_dump_restore($oldnode, %node_params);
+ }
}
# Initialize a new node for the upgrade.
@@ -539,4 +555,128 @@ my $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
compare_files($dump1_filtered, $dump2_filtered,
'old and new dumps match after pg_upgrade');
+# Test dump and restore of objects left behind by the regression run.
+#
+# It is expected that regression tests, which create `regression` database, are
+# run on `src_node`, which in turn, is left in running state. A fresh node is
+# created using given `node_params`, which are expected to be the same ones used
+# to create `src_node`, so as to avoid any differences in the databases.
+#
+# Plain dumps from both the nodes are compared to make sure that all the dumped
+# objects are restored faithfully.
+sub test_regression_dump_restore
+{
+ my ($src_node, %node_params) = @_;
+ my $dst_node = PostgreSQL::Test::Cluster->new('dst_node');
+
+ # Make sure that the source and destination nodes have the same version and
+ # do not use custom install paths. In both the cases, the dump files may
+ # require additional adjustments unknown to code here. Do not run this test
+ # in such a case to avoid utilizing the time and resources unnecessarily.
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default installation");
+ return;
+ }
+
+ # Dump the original database for comparison later.
+ my $src_dump =
+ get_dump_for_comparison($src_node, 'regression', 'src_dump', 1);
+
+ # Setup destination database cluster
+ $dst_node->init(%node_params);
+ # Stabilize stats for comparison.
+ $dst_node->append_conf('postgresql.conf', 'autovacuum = off');
+ $dst_node->start;
+
+ # Test all formats one by one.
+ for my $format ('plain', 'tar', 'directory', 'custom')
+ {
+ my $dump_file = "$tempdir/regression_dump.$format";
+ my $restored_db = 'regression_' . $format;
+
+ # Use --create in dump and restore commands so that the restored
+ # database has the same configurable variable settings as the original
+ # database and the plain dumps taken for comparsion do not differ
+ # because of locale changes. Additionally this provides test coverage
+ # for --create option.
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '--create',
+ '-d', 'postgres', $dump_file
+ ];
+ }
+ $dst_node->command_ok(@restore_command,
+ "restored dump taken in $format format on destination instance");
+
+ my $dst_dump =
+ get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump.' . $format, 0);
+
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database (using $format format) match"
+ );
+
+ # Rename the restored database so that it is available for debugging in
+ # case the test fails.
+ $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
+ }
+}
+
+# Dump database `db` from the given `node` in plain format and adjust it for
+# comparing dumps from the original and the restored database.
+#
+# `file_prefix` is used to create unique names for all dump files so that they
+# remain available for debugging in case the test fails.
+#
+# `adjust_child_columns` is passed to adjust_regress_dumpfile() which actually
+# adjusts the dump output.
+#
+# The name of the file containting adjusted dump is returned.
+sub get_dump_for_comparison
+{
+ my ($node, $db, $file_prefix, $adjust_child_columns) = @_;
+
+ my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
+ my $dump_adjusted = "${dumpfile}_adjusted";
+
+ # Usually we avoid comparing statistics in our tests since it is flaky by
+ # nature. However, if statistics is dumped and restored it is expected to be
+ # restored as it is i.e. the statistics from the original database and that
+ # from the restored database should match. We turn off autovacuum on the
+ # source and the target database to avoid any statistics update during
+ # restore operation. Hence we do not exclude statistics from dump.
+ $node->command_ok(
+ [
+ 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ $dumpfile
+ ],
+ 'dump for comparison succeeded');
+
+ open(my $dh, '>', $dump_adjusted)
+ || die
+ "could not open $dump_adjusted for writing the adjusted dump: $!";
+ print $dh adjust_regress_dumpfile(slurp_file($dumpfile),
+ $adjust_child_columns);
+ close($dh);
+
+ return $dump_adjusted;
+}
+
done_testing();
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index d82fb67540e..def89650ead 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustUpgrade.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustDump.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
uninstall:
@@ -36,6 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
endif
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..74b9a60cf34
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,167 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustDump - helper module for dump and restore tests
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::AdjustDump;
+
+ # Adjust contents of dump output file so that dump output from original
+ # regression database and that from the restored regression database match
+ $dump = adjust_regress_dumpfile($dump, $adjust_child_columns);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustDump> encapsulates various hacks needed to
+compare the results of dump and restore tests
+
+=cut
+
+package PostgreSQL::Test::AdjustDump;
+
+use strict;
+use warnings FATAL => 'all';
+
+use Exporter 'import';
+use Test::More;
+
+our @EXPORT = qw(
+ adjust_regress_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $dump = adjust_regress_dumpfile($dump, $adjust_child_columns)
+
+If we take dump of the regression database left behind after running regression
+tests, restore the dump, and take dump of the restored regression database, the
+outputs of both the dumps differ in the following cases. This routine adjusts
+the given dump so that dump outputs from the original and restored database,
+respectively, match.
+
+Case 1: Some regression tests purposefully create child tables in such a way
+that the order of their inherited columns differ from column orders of their
+respective parents. In the restored database, however, the order of their
+inherited columns are same as that of their respective parents. Thus the column
+orders of these child tables in the original database and those in the restored
+database differ, causing difference in the dump outputs. See MergeAttributes()
+and dumpTableSchema() for details. This routine rearranges the column
+declarations in the relevant C<CREATE TABLE... INHERITS> statements in the dump
+file from original database to match those from the restored database. We could,
+instead, adjust the statements in the dump from the restored database to match
+those from original database or adjust both to a canonical order. But we have
+chosen to adjust the statements in the dump from original database for no
+particular reason.
+
+Case 2: When dumping COPY statements the columns are ordered by their attribute
+number by fmtCopyColumnList(). If a column is added to a parent table after a
+child has inherited the parent and the child has its own columns, the attribute
+number of the column changes after restoring the child table. This is because
+when executing the dumped C<CREATE TABLE... INHERITS> statement all the parent
+attributes are created before any child attributes. Thus the order of columns in
+COPY statements dumped from the original and the restored databases,
+respectively, differs. Such tables in regression tests are listed below. It is
+hard to adjust the column order in the COPY statement along with the data. Hence
+we just remove such COPY statements from the dump output.
+
+Additionally the routine adjusts blank and new lines to avoid noise.
+
+Note: Usually we avoid comparing statistics in our tests since it is flaky by
+nature. However, if statistics is dumped and restored it is expected to be
+restored as it is i.e. the statistics from the original database and that from
+the restored database should match. Hence we do not filter statistics from dump,
+if it's dumped.
+
+Arguments:
+
+=over
+
+=item C<dump>: Contents of dump file
+
+=item C<adjust_child_columns>: 1 indicates that the given dump file requires
+adjusting columns in the child tables; usually when the dump is from original
+database. 0 indicates no such adjustment is needed; usually when the dump is
+from restored database.
+
+=back
+
+Returns the adjusted dump text.
+
+=cut
+
+sub adjust_regress_dumpfile
+{
+ my ($dump, $adjust_child_columns) = @_;
+
+ # use Unix newlines
+ $dump =~ s/\r\n/\n/g;
+
+ # Adjust the CREATE TABLE ... INHERITS statements.
+ if ($adjust_child_columns)
+ {
+ my $saved_dump = $dump;
+
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_stored_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_stored_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_virtual_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_virtual_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c1\s\()
+ (\n\s+int_four\sbigint),
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint)/$1$4,$2,$3/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c1 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c2\s\()
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint),
+ (\n\s+int_four\sbigint)/$1$3,$4,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c2 adjustments');
+ }
+
+ # Remove COPY statements with differing column order
+ for my $table (
+ 'public\.b_star', 'public\.c_star',
+ 'public\.cc2', 'public\.d_star',
+ 'public\.e_star', 'public\.f_star',
+ 'public\.renamecolumnanother', 'public\.renamecolumnchild',
+ 'public\.test_type_diff2_c1', 'public\.test_type_diff2_c2',
+ 'public\.test_type_diff_c')
+ {
+ $dump =~ s/^COPY\s$table\s\(.+?^\\\.$//sm;
+ }
+
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
+ return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff --git a/src/test/perl/meson.build b/src/test/perl/meson.build
index 58e30f15f9d..492ca571ff8 100644
--- a/src/test/perl/meson.build
+++ b/src/test/perl/meson.build
@@ -14,4 +14,5 @@ install_data(
'PostgreSQL/Test/Cluster.pm',
'PostgreSQL/Test/BackgroundPsql.pm',
'PostgreSQL/Test/AdjustUpgrade.pm',
+ 'PostgreSQL/Test/AdjustDump.pm',
install_dir: dir_pgxs / 'src/test/perl/PostgreSQL/Test')
base-commit: 190dc27998d5b7b4c36e12bebe62f7176f4b4507
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-20 15:06 ` vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 11:51 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: vignesh C @ 2025-03-20 15:06 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 19 Mar 2025 at 17:13, Ashutosh Bapat
<[email protected]> wrote:
>
> On Thu, Mar 13, 2025 at 6:10 PM Ashutosh Bapat
> <[email protected]> wrote:
> > >
> > > I think the fix is to explicitly pass --lc-monetary to the old cluster
> > > and the restored cluster. 003 patch in the attached patch set does
> > > that. Please check if it fixes the issue for you.
> > >
> > > Additionally we should check that it gets copied to the new cluster as
> > > well. But I haven't figured out how to get those settings yet. This
> > > treatment is similar to how --lc-collate and --lc-ctype are treated. I
> > > am wondering whether we should explicitly pass --lc-messages,
> > > --lc-time and --lc-numeric as well.
> > >
> > > 2d819a08a1cbc11364e36f816b02e33e8dcc030b introduced buildin locale
> > > provider and added overrides to LC_COLLATE and LC_TYPE. But it did not
> > > override other LC_, which I think it should have. In pure upgrade
> > > test, the upgraded node inherits the locale settings of the original
> > > cluster, so this wasn't apparent. But with pg_dump testing, the
> > > original and restored databases are independent. Hence I think we have
> > > to override all LC_* settings by explicitly mentioning --lc-* options
> > > to initdb. Please let me know what you think about this?
> > >
>
> Investigated this further. The problem is that the pg_regress run
> creates regression database with specific properties but the restored
> database does not have those properties. That led me to a better
> solution. Additionally it's local to the new test. Use --create when
> dumping and restoring the regression database. This way the database
> properties or "configuration variable settings (as pg_dump
> documentation calls them) are copied to the restored database as well.
> Those properties include LC_MONETARY. Additionally now the test covers
> --create option as well.
>
> PFA patches.
Will it help the execution time if we use --jobs in case of pg_dump
and pg_restore wherever supported:
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [ 'psql', '-d', 'postgres',
'-f', $dump_file ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '--create',
+ '-d', 'postgres', $dump_file
+ ];
+ }
Should the copyright be only 2025 in this case:
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm
b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..74b9a60cf34
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,167 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
Regards,
Vignesh
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
@ 2025-03-20 16:39 ` Alvaro Herrera <[email protected]>
2025-03-21 12:45 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
1 sibling, 2 replies; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-20 16:39 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-20, vignesh C wrote:
> Will it help the execution time if we use --jobs in case of pg_dump
> and pg_restore wherever supported:
As I said in another thread, I think we should enable this test to run
without requiring any PG_TEST_EXTRA, because otherwise the only way to
know about problems is to commit a patch and wait for buildfarm to run
it. Furthermore, I think running all 4 dump format modes is a waste of
time; there isn't any extra coverage by running this test in additional
formats.
Putting those two thoughts together with yours about running with -j,
I propose that what we should do is make this test use -Fc with no
compression (to avoid wasting CPU on that) and use a lowish -j value for
both pg_dump and pg_restore, probably 2, or 3 at most. (Not more,
because this is likely to run in parallel with other tests anyway.)
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"No renuncies a nada. No te aferres a nada."
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-21 12:45 ` vignesh C <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: vignesh C @ 2025-03-21 12:45 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, 20 Mar 2025 at 22:09, Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-20, vignesh C wrote:
>
> > Will it help the execution time if we use --jobs in case of pg_dump
> > and pg_restore wherever supported:
>
> As I said in another thread, I think we should enable this test to run
> without requiring any PG_TEST_EXTRA, because otherwise the only way to
> know about problems is to commit a patch and wait for buildfarm to run
> it. Furthermore, I think running all 4 dump format modes is a waste of
> time; there isn't any extra coverage by running this test in additional
> formats.
+1 for running it in only one of the formats.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-21 13:09 ` Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-21 13:09 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 20, 2025 at 10:09 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-20, vignesh C wrote:
>
> > Will it help the execution time if we use --jobs in case of pg_dump
> > and pg_restore wherever supported:
>
> As I said in another thread, I think we should enable this test to run
> without requiring any PG_TEST_EXTRA, because otherwise the only way to
> know about problems is to commit a patch and wait for buildfarm to run
> it. Furthermore, I think running all 4 dump format modes is a waste of
> time; there isn't any extra coverage by running this test in additional
> formats.
>
> Putting those two thoughts together with yours about running with -j,
> I propose that what we should do is make this test use -Fc with no
> compression (to avoid wasting CPU on that) and use a lowish -j value for
> both pg_dump and pg_restore, probably 2, or 3 at most. (Not more,
> because this is likely to run in parallel with other tests anyway.)
-Fc and -j are not allowed. -j is only allowed for directory format.
$ pg_dump -Fc -j2
pg_dump: error: parallel backup only supported by the directory format
Using just directory format, on my laptop with dev build (because
that's what most developers will use when running the tests)
$ meson test -C $BuildDir pg_upgrade/002_pg_upgrade | grep 002_pg_upgrade
without dump/restore test
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
33.51s 19 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
34.22s 19 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
34.64s 19 subtests passed
without -j, extra ~9 seconds
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
43.33s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
43.25s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
43.10s 28 subtests passed
with -j2, extra 7.5 seconds
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
42.77s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.67s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.88s 28 subtests passed
with -j3, extra 7 seconds
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
40.77s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.05s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.28s 28 subtests passed
Between -j2 and -j3 there's not much difference so we could use -j2.
But it still takes 7.5 extra seconds which almost 20% extra time. Do
you think that will be acceptable? I saw somewhere Andres mentioning
that he runs this test quite frequently. Please note that I would very
much like this test to be run by default, but Tom Lane has expressed a
concern about adding even that much time [1] to run the test and
mentioned that he would like the test to be opt-in.
When I started writing the test one year before, people raised
concerns about how useful the test would be. Within a year it has
shown 4 bugs. I have similar feeling about the formats - it's doubtful
now but will prove useful soon especially with the work happening on
dump formats in nearby threads. If we run the test by default, we
could run directory with -j by default and leave other formats as
opt-in OR just forget those formats for now. But If we are going to
make it opt-in, testing all formats gives the extra coverage.
About the format coverage, consensus so far is me and Daniel are for
including all formats when running test as opt-in. Alvaro and Vignesh
are for just one format. We need a tie-breaker or someone amongst us
needs to change their vote :D.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-21 14:39 ` Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-21 14:39 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
I passed PROVE_FLAGS="--timer -v" to get the timings and run under
--format=directory.
Without new test:
ok 23400 ms ( 0.00 usr 0.00 sys + 2.84 cusr 1.53 csys = 4.37 CPU)
ok 23409 ms ( 0.00 usr 0.01 sys + 2.81 cusr 1.53 csys = 4.35 CPU)
With new test, under --format=directory:
-j2 (parallel, default gzip compression)
ok 27517 ms ( 0.00 usr 0.00 sys + 3.92 cusr 1.86 csys = 5.78 CPU)
ok 27772 ms ( 0.01 usr 0.00 sys + 3.96 cusr 1.86 csys = 5.83 CPU)
ok 27654 ms ( 0.00 usr 0.00 sys + 3.81 cusr 1.94 csys = 5.75 CPU)
ok 27663 ms ( 0.00 usr 0.00 sys + 4.11 cusr 1.71 csys = 5.82 CPU)
-j2 --compress=0
ok 27710 ms ( 0.00 usr 0.00 sys + 3.79 cusr 1.86 csys = 5.65 CPU)
ok 27567 ms ( 0.01 usr 0.00 sys + 3.67 cusr 1.96 csys = 5.64 CPU)
ok 27582 ms ( 0.00 usr 0.00 sys + 3.60 cusr 1.90 csys = 5.50 CPU)
ok 27519 ms ( 0.01 usr 0.00 sys + 3.71 cusr 1.80 csys = 5.52 CPU)
-j2 --compress=zstd
ok 27240 ms ( 0.01 usr 0.00 sys + 3.65 cusr 2.10 csys = 5.76 CPU)
ok 27301 ms ( 0.01 usr 0.00 sys + 3.77 cusr 1.97 csys = 5.75 CPU)
-j2 --compress=zstd:1
ok 27695 ms ( 0.01 usr 0.00 sys + 3.66 cusr 2.05 csys = 5.72 CPU)
ok 27671 ms ( 0.01 usr 0.00 sys + 3.76 cusr 1.95 csys = 5.72 CPU)
--compress=zstd:1 (no parallelism)
ok 28417 ms ( 0.01 usr 0.00 sys + 3.90 cusr 1.75 csys = 5.66 CPU)
ok 28388 ms ( 0.00 usr 0.00 sys + 3.74 cusr 1.81 csys = 5.55 CPU)
--compress=zstd (no parallelism)
ok 28310 ms ( 0.00 usr 0.01 sys + 3.81 cusr 1.83 csys = 5.65 CPU)
ok 28277 ms ( 0.01 usr 0.00 sys + 3.71 cusr 1.87 csys = 5.59 CPU)
So apparently, zstd if available is a bit better than gzip and
parallelism is better than no. But the differences are small -- half a
second or so. The total increase in runtime in the best case is about
four seconds. In all cases I used the same parallelism in pg_restore
than pg_dump; not sure if that could cause a difference.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-21 16:03 ` Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-21 16:03 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 21, 2025 at 8:13 PM Alvaro Herrera <[email protected]> wrote:
>
> I passed PROVE_FLAGS="--timer -v" to get the timings and run under
> --format=directory.
>
> Without new test:
> ok 23400 ms ( 0.00 usr 0.00 sys + 2.84 cusr 1.53 csys = 4.37 CPU)
> ok 23409 ms ( 0.00 usr 0.01 sys + 2.81 cusr 1.53 csys = 4.35 CPU)
>
>
> With new test, under --format=directory:
> -j2 (parallel, default gzip compression)
> ok 27517 ms ( 0.00 usr 0.00 sys + 3.92 cusr 1.86 csys = 5.78 CPU)
> ok 27772 ms ( 0.01 usr 0.00 sys + 3.96 cusr 1.86 csys = 5.83 CPU)
> ok 27654 ms ( 0.00 usr 0.00 sys + 3.81 cusr 1.94 csys = 5.75 CPU)
> ok 27663 ms ( 0.00 usr 0.00 sys + 4.11 cusr 1.71 csys = 5.82 CPU)
>
> -j2 --compress=0
> ok 27710 ms ( 0.00 usr 0.00 sys + 3.79 cusr 1.86 csys = 5.65 CPU)
> ok 27567 ms ( 0.01 usr 0.00 sys + 3.67 cusr 1.96 csys = 5.64 CPU)
> ok 27582 ms ( 0.00 usr 0.00 sys + 3.60 cusr 1.90 csys = 5.50 CPU)
> ok 27519 ms ( 0.01 usr 0.00 sys + 3.71 cusr 1.80 csys = 5.52 CPU)
>
> -j2 --compress=zstd
> ok 27240 ms ( 0.01 usr 0.00 sys + 3.65 cusr 2.10 csys = 5.76 CPU)
> ok 27301 ms ( 0.01 usr 0.00 sys + 3.77 cusr 1.97 csys = 5.75 CPU)
>
> -j2 --compress=zstd:1
> ok 27695 ms ( 0.01 usr 0.00 sys + 3.66 cusr 2.05 csys = 5.72 CPU)
> ok 27671 ms ( 0.01 usr 0.00 sys + 3.76 cusr 1.95 csys = 5.72 CPU)
>
> --compress=zstd:1 (no parallelism)
> ok 28417 ms ( 0.01 usr 0.00 sys + 3.90 cusr 1.75 csys = 5.66 CPU)
> ok 28388 ms ( 0.00 usr 0.00 sys + 3.74 cusr 1.81 csys = 5.55 CPU)
>
> --compress=zstd (no parallelism)
> ok 28310 ms ( 0.00 usr 0.01 sys + 3.81 cusr 1.83 csys = 5.65 CPU)
> ok 28277 ms ( 0.01 usr 0.00 sys + 3.71 cusr 1.87 csys = 5.59 CPU)
>
>
> So apparently, zstd if available is a bit better than gzip and
> parallelism is better than no. But the differences are small -- half a
> second or so. The total increase in runtime in the best case is about
> four seconds. In all cases I used the same parallelism in pg_restore
> than pg_dump; not sure if that could cause a difference.
I used the same parallelism in pg_restore and pg_dump too. And your
numbers seem to be similar to mine; slightly less than 20% slowdown.
But is that slowdown acceptable? From the earlier discussions, it
seems the answer is No. Haven't heard otherwise.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-21 18:07 ` Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-21 18:07 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-21, Ashutosh Bapat wrote:
> I used the same parallelism in pg_restore and pg_dump too. And your
> numbers seem to be similar to mine; slightly less than 20% slowdown.
> But is that slowdown acceptable? From the earlier discussions, it
> seems the answer is No. Haven't heard otherwise.
I don't think we need to see slowdown this in relative terms, the way we
would discuss a change in the executor. This is not a change that
would affect user-level stuff in any way. We need to see it in absolute
terms: in machines similar to mine, the pg_upgrade test would go from
taking 23s to taking 27s. This is 4s slower, but this isn't an increase
in total test runtime, because decently run test suites run multiple
tests in parallel. This is the same that Peter said in [1]. The total
test runtime change might not be *that* large. I'll take a few numbers
and report back.
[1] https://postgr.es/m/[email protected]
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I love the Postgres community. It's all about doing things _properly_. :-)"
(David Garamond)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-24 09:54 ` Ashutosh Bapat <[email protected]>
2025-03-24 09:59 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-24 09:54 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 21, 2025 at 11:38 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-21, Ashutosh Bapat wrote:
>
> > I used the same parallelism in pg_restore and pg_dump too. And your
> > numbers seem to be similar to mine; slightly less than 20% slowdown.
> > But is that slowdown acceptable? From the earlier discussions, it
> > seems the answer is No. Haven't heard otherwise.
>
> I don't think we need to see slowdown this in relative terms, the way we
> would discuss a change in the executor. This is not a change that
> would affect user-level stuff in any way. We need to see it in absolute
> terms: in machines similar to mine, the pg_upgrade test would go from
> taking 23s to taking 27s. This is 4s slower, but this isn't an increase
> in total test runtime, because decently run test suites run multiple
> tests in parallel. This is the same that Peter said in [1]. The total
> test runtime change might not be *that* large. I'll take a few numbers
> and report back.
Using -j2 in pg_dump and -j3 in pg_restore does not improve timing
much on my laptop. I have used -j2 for both pg_dump and restore
instead of -j3 so as to avoid using more cores when tests are run in
parallel.
Further to reduce run time, I tried -1/--single-transaction but that's
not allowed with --create. I also tried --transaction-size=1000 but
that doesn't affect the run time of the test. Next I thought of using
standard output and input instead of files but it doesn't help since
1. directory format cannot use those and it's the only format allowing
parallelism, 2. that's slower than using files with --no-sync. Didn't
find any other way which can help us reduce the test time.
Please note that the dumps taken for comparison cannot use -j since
they are required to be in "plain" format so that text manipulation
comparison works on them.
One concern I have with directory format is the dumped database is not
readable. This might make investigating a but identified the test a
bit more complex. But I guess, in such a case investigator can either
use the dumps taken for comparison or change the code to use plain
format for investigation. So it's a price we pay for making test
faster.
Here's next patchset:
0001 - it's the same 0001 patch as previous one, includes the test
with all formats and also the PG_TEST_EXTRA option
0002 - removes PG_TEST_EXTRA and also tests only one format
--directory with -j2 with default compression. It should be merged
into 0001 before committing. This is a separate patch for now in case
we decide to go back to 0001.
0003 - same as 0002 in the previous patch set. It excludes statistics
from comparison, otherwise the test will fail because of bug reported
at [1]. Ideally we shouldn't commit this patch so as to test
statistics dump and restore, but in case we need the test to pass till
the bug is fixed, we should merge this patch to 0001 before
committing.
[1] https://www.postgresql.org/message-id/[email protected]...
--
Best Wishes,
Ashutosh Bapat
Attachments:
[text/x-patch] 0001-Test-pg_dump-restore-of-regression-objects-20250324.patch (16.5K, ../../CAExHW5vw_KaZrjWSNJx-QHF12D4KCmV=AAii3Zh3RHmY43gesw@mail.gmail.com/2-0001-Test-pg_dump-restore-of-regression-objects-20250324.patch)
download | inline diff:
From fcfd0d25ecd374d55970817b4d3ea2aecdd58251 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 27 Jun 2024 10:03:53 +0530
Subject: [PATCH 1/3] Test pg_dump/restore of regression objects
002_pg_upgrade.pl tests pg_upgrade of the regression database left
behind by regression run. Modify it to test dump and restore of the
regression database as well.
Regression database created by regression run contains almost all the
database objects supported by PostgreSQL in various states. Hence the
new testcase covers dump and restore scenarios not covered by individual
dump/restore cases. Till now 002_pg_upgrade only tested dump/restore
through pg_upgrade which only uses binary mode. Many regression tests
mention that they leave objects behind for dump/restore testing but they
are not tested in a non-binary mode. The new testcase closes that
gap.
Testing dump and restore of regression database makes this test run
longer for a relatively smaller benefit. Hence run it only when
explicitly requested by user by specifying "regress_dump_test" in
PG_TEST_EXTRA.
Note For the reviewers:
The new test has uncovered many bugs so far in one year.
1. Introduced by 14e87ffa5c54. Fixed in fd41ba93e4630921a72ed5127cd0d552a8f3f8fc.
2. Introduced by 0413a556990ba628a3de8a0b58be020fd9a14ed0. Reverted in 74563f6b90216180fc13649725179fc119dddeb5.
3. Fixed by d611f8b1587b8f30caa7c0da99ae5d28e914d54f
3. Being discussed on hackers at https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Author: Ashutosh Bapat
Reviewed by: Michael Pacquire, Daniel Gustafsson, Tom Lane, Alvaro Herrera
Discussion: https://www.postgresql.org/message-id/CAExHW5uF5V=Cjecx3_Z=7xfh4rg2Wf61PT+hfquzjBqouRzQJQ@mail.gmail.com
---
doc/src/sgml/regress.sgml | 12 ++
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 144 ++++++++++++++++-
src/test/perl/Makefile | 2 +
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 167 ++++++++++++++++++++
src/test/perl/meson.build | 1 +
5 files changed, 324 insertions(+), 2 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/AdjustDump.pm
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 0e5e8e8f309..237b974b3ab 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,6 +357,18 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>regress_dump_test</literal></term>
+ <listitem>
+ <para>
+ When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
+ tests dump and restore of regression database left behind by the
+ regression run. Not enabled by default because it is time and resource
+ consuming.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 00051b85035..d08eea6693f 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -12,6 +12,7 @@ use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::AdjustUpgrade;
+use PostgreSQL::Test::AdjustDump;
use Test::More;
# Can be changed to test the other modes.
@@ -35,8 +36,8 @@ sub generate_db
"created database with ASCII characters from $from_char to $to_char");
}
-# Filter the contents of a dump before its use in a content comparison.
-# This returns the path to the filtered dump.
+# Filter the contents of a dump before its use in a content comparison for
+# upgrade testing. This returns the path to the filtered dump.
sub filter_dump
{
my ($is_old, $old_version, $dump_file) = @_;
@@ -262,6 +263,21 @@ else
}
}
is($rc, 0, 'regression tests pass');
+
+ # Test dump/restore of the objects left behind by regression. Ideally it
+ # should be done in a separate TAP test, but doing it here saves us one full
+ # regression run.
+ #
+ # This step takes several extra seconds and some extra disk space, so
+ # requires an opt-in with the PG_TEST_EXTRA environment variable.
+ #
+ # Do this while the old cluster is running before it is shut down by the
+ # upgrade test.
+ if ( $ENV{PG_TEST_EXTRA}
+ && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
+ {
+ test_regression_dump_restore($oldnode, %node_params);
+ }
}
# Initialize a new node for the upgrade.
@@ -539,4 +555,128 @@ my $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
compare_files($dump1_filtered, $dump2_filtered,
'old and new dumps match after pg_upgrade');
+# Test dump and restore of objects left behind by the regression run.
+#
+# It is expected that regression tests, which create `regression` database, are
+# run on `src_node`, which in turn, is left in running state. A fresh node is
+# created using given `node_params`, which are expected to be the same ones used
+# to create `src_node`, so as to avoid any differences in the databases.
+#
+# Plain dumps from both the nodes are compared to make sure that all the dumped
+# objects are restored faithfully.
+sub test_regression_dump_restore
+{
+ my ($src_node, %node_params) = @_;
+ my $dst_node = PostgreSQL::Test::Cluster->new('dst_node');
+
+ # Make sure that the source and destination nodes have the same version and
+ # do not use custom install paths. In both the cases, the dump files may
+ # require additional adjustments unknown to code here. Do not run this test
+ # in such a case to avoid utilizing the time and resources unnecessarily.
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default installation");
+ return;
+ }
+
+ # Dump the original database for comparison later.
+ my $src_dump =
+ get_dump_for_comparison($src_node, 'regression', 'src_dump', 1);
+
+ # Setup destination database cluster
+ $dst_node->init(%node_params);
+ # Stabilize stats for comparison.
+ $dst_node->append_conf('postgresql.conf', 'autovacuum = off');
+ $dst_node->start;
+
+ # Test all formats one by one.
+ for my $format ('plain', 'tar', 'directory', 'custom')
+ {
+ my $dump_file = "$tempdir/regression_dump.$format";
+ my $restored_db = 'regression_' . $format;
+
+ # Use --create in dump and restore commands so that the restored
+ # database has the same configurable variable settings as the original
+ # database and the plain dumps taken for comparsion do not differ
+ # because of locale changes. Additionally this provides test coverage
+ # for --create option.
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '--create',
+ '-d', 'postgres', $dump_file
+ ];
+ }
+ $dst_node->command_ok(@restore_command,
+ "restored dump taken in $format format on destination instance");
+
+ my $dst_dump =
+ get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump.' . $format, 0);
+
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database (using $format format) match"
+ );
+
+ # Rename the restored database so that it is available for debugging in
+ # case the test fails.
+ $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
+ }
+}
+
+# Dump database `db` from the given `node` in plain format and adjust it for
+# comparing dumps from the original and the restored database.
+#
+# `file_prefix` is used to create unique names for all dump files so that they
+# remain available for debugging in case the test fails.
+#
+# `adjust_child_columns` is passed to adjust_regress_dumpfile() which actually
+# adjusts the dump output.
+#
+# The name of the file containting adjusted dump is returned.
+sub get_dump_for_comparison
+{
+ my ($node, $db, $file_prefix, $adjust_child_columns) = @_;
+
+ my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
+ my $dump_adjusted = "${dumpfile}_adjusted";
+
+ # Usually we avoid comparing statistics in our tests since it is flaky by
+ # nature. However, if statistics is dumped and restored it is expected to be
+ # restored as it is i.e. the statistics from the original database and that
+ # from the restored database should match. We turn off autovacuum on the
+ # source and the target database to avoid any statistics update during
+ # restore operation. Hence we do not exclude statistics from dump.
+ $node->command_ok(
+ [
+ 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ $dumpfile
+ ],
+ 'dump for comparison succeeded');
+
+ open(my $dh, '>', $dump_adjusted)
+ || die
+ "could not open $dump_adjusted for writing the adjusted dump: $!";
+ print $dh adjust_regress_dumpfile(slurp_file($dumpfile),
+ $adjust_child_columns);
+ close($dh);
+
+ return $dump_adjusted;
+}
+
done_testing();
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index d82fb67540e..def89650ead 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustUpgrade.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustDump.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
uninstall:
@@ -36,6 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
endif
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..74b9a60cf34
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,167 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustDump - helper module for dump and restore tests
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::AdjustDump;
+
+ # Adjust contents of dump output file so that dump output from original
+ # regression database and that from the restored regression database match
+ $dump = adjust_regress_dumpfile($dump, $adjust_child_columns);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustDump> encapsulates various hacks needed to
+compare the results of dump and restore tests
+
+=cut
+
+package PostgreSQL::Test::AdjustDump;
+
+use strict;
+use warnings FATAL => 'all';
+
+use Exporter 'import';
+use Test::More;
+
+our @EXPORT = qw(
+ adjust_regress_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $dump = adjust_regress_dumpfile($dump, $adjust_child_columns)
+
+If we take dump of the regression database left behind after running regression
+tests, restore the dump, and take dump of the restored regression database, the
+outputs of both the dumps differ in the following cases. This routine adjusts
+the given dump so that dump outputs from the original and restored database,
+respectively, match.
+
+Case 1: Some regression tests purposefully create child tables in such a way
+that the order of their inherited columns differ from column orders of their
+respective parents. In the restored database, however, the order of their
+inherited columns are same as that of their respective parents. Thus the column
+orders of these child tables in the original database and those in the restored
+database differ, causing difference in the dump outputs. See MergeAttributes()
+and dumpTableSchema() for details. This routine rearranges the column
+declarations in the relevant C<CREATE TABLE... INHERITS> statements in the dump
+file from original database to match those from the restored database. We could,
+instead, adjust the statements in the dump from the restored database to match
+those from original database or adjust both to a canonical order. But we have
+chosen to adjust the statements in the dump from original database for no
+particular reason.
+
+Case 2: When dumping COPY statements the columns are ordered by their attribute
+number by fmtCopyColumnList(). If a column is added to a parent table after a
+child has inherited the parent and the child has its own columns, the attribute
+number of the column changes after restoring the child table. This is because
+when executing the dumped C<CREATE TABLE... INHERITS> statement all the parent
+attributes are created before any child attributes. Thus the order of columns in
+COPY statements dumped from the original and the restored databases,
+respectively, differs. Such tables in regression tests are listed below. It is
+hard to adjust the column order in the COPY statement along with the data. Hence
+we just remove such COPY statements from the dump output.
+
+Additionally the routine adjusts blank and new lines to avoid noise.
+
+Note: Usually we avoid comparing statistics in our tests since it is flaky by
+nature. However, if statistics is dumped and restored it is expected to be
+restored as it is i.e. the statistics from the original database and that from
+the restored database should match. Hence we do not filter statistics from dump,
+if it's dumped.
+
+Arguments:
+
+=over
+
+=item C<dump>: Contents of dump file
+
+=item C<adjust_child_columns>: 1 indicates that the given dump file requires
+adjusting columns in the child tables; usually when the dump is from original
+database. 0 indicates no such adjustment is needed; usually when the dump is
+from restored database.
+
+=back
+
+Returns the adjusted dump text.
+
+=cut
+
+sub adjust_regress_dumpfile
+{
+ my ($dump, $adjust_child_columns) = @_;
+
+ # use Unix newlines
+ $dump =~ s/\r\n/\n/g;
+
+ # Adjust the CREATE TABLE ... INHERITS statements.
+ if ($adjust_child_columns)
+ {
+ my $saved_dump = $dump;
+
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_stored_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_stored_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_virtual_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_virtual_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c1\s\()
+ (\n\s+int_four\sbigint),
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint)/$1$4,$2,$3/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c1 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c2\s\()
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint),
+ (\n\s+int_four\sbigint)/$1$3,$4,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c2 adjustments');
+ }
+
+ # Remove COPY statements with differing column order
+ for my $table (
+ 'public\.b_star', 'public\.c_star',
+ 'public\.cc2', 'public\.d_star',
+ 'public\.e_star', 'public\.f_star',
+ 'public\.renamecolumnanother', 'public\.renamecolumnchild',
+ 'public\.test_type_diff2_c1', 'public\.test_type_diff2_c2',
+ 'public\.test_type_diff_c')
+ {
+ $dump =~ s/^COPY\s$table\s\(.+?^\\\.$//sm;
+ }
+
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
+ return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff --git a/src/test/perl/meson.build b/src/test/perl/meson.build
index 58e30f15f9d..492ca571ff8 100644
--- a/src/test/perl/meson.build
+++ b/src/test/perl/meson.build
@@ -14,4 +14,5 @@ install_data(
'PostgreSQL/Test/Cluster.pm',
'PostgreSQL/Test/BackgroundPsql.pm',
'PostgreSQL/Test/AdjustUpgrade.pm',
+ 'PostgreSQL/Test/AdjustDump.pm',
install_dir: dir_pgxs / 'src/test/perl/PostgreSQL/Test')
base-commit: 73eba5004a06a744b6b8570e42432b9e9f75997b
--
2.34.1
[text/x-patch] 0003-Do-not-dump-statistics-in-the-file-dumped-f-20250324.patch (2.1K, ../../CAExHW5vw_KaZrjWSNJx-QHF12D4KCmV=AAii3Zh3RHmY43gesw@mail.gmail.com/3-0003-Do-not-dump-statistics-in-the-file-dumped-f-20250324.patch)
download | inline diff:
From 435c659489b34a803675abb65144fab6f0550432 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Tue, 25 Feb 2025 11:42:51 +0530
Subject: [PATCH 3/3] Do not dump statistics in the file dumped for comparison
The dumped and restored statistics of a materialized view may differ as
reported in [1]. Hence do not dump the statistics to avoid differences
in the dump output from the original and restored database.
[1] https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Ashutosh Bapat
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index cbd9831bf9e..abe93a49258 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -630,15 +630,15 @@ sub get_dump_for_comparison
my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
my $dump_adjusted = "${dumpfile}_adjusted";
- # Usually we avoid comparing statistics in our tests since it is flaky by
- # nature. However, if statistics is dumped and restored it is expected to be
- # restored as it is i.e. the statistics from the original database and that
- # from the restored database should match. We turn off autovacuum on the
- # source and the target database to avoid any statistics update during
- # restore operation. Hence we do not exclude statistics from dump.
+ # If statistics is dumped and restored it is expected to be restored as it
+ # is i.e. the statistics from the original database and that from the
+ # restored database should match. We turn off autovacuum on the source and
+ # the target database to avoid any statistics update during restore
+ # operation. But as of now, there are cases when statistics is not being
+ # restored faithfully. Hence for now do not dump statistics.
$node->command_ok(
[
- 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ 'pg_dump', '--no-sync', '--no-statistics', '-d', $node->connstr($db), '-f',
$dumpfile
],
'dump for comparison succeeded');
--
2.34.1
[text/x-patch] 0002-Use-only-one-format-and-make-the-test-run-d-20250324.patch (5.4K, ../../CAExHW5vw_KaZrjWSNJx-QHF12D4KCmV=AAii3Zh3RHmY43gesw@mail.gmail.com/4-0002-Use-only-one-format-and-make-the-test-run-d-20250324.patch)
download | inline diff:
From f26f88364a196dc9589ca451cb54f5e514e3422e Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Mon, 24 Mar 2025 11:21:12 +0530
Subject: [PATCH 2/3] Use only one format and make the test run default
According to Alvaro (and I agree with him), the test should be run by
default. Otherwise we get to know about a bug only after buildfarm
animal where it's enabled reports a failure. Further testing only one
format may suffice; since all the formats have shown the same bugs till
now.
If we use --directory format we can use -j which reduces the time taken
by dump/restore test by about 12%.
This patch removes PG_TEST_EXTRA option as well as runs the test only in
directory format with parallelism enabled.
Note for committer: If we decide to accept this change, it should be
merged with the previous commit.
---
doc/src/sgml/regress.sgml | 12 ----
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 76 +++++++++-----------------
2 files changed, 25 insertions(+), 63 deletions(-)
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 237b974b3ab..0e5e8e8f309 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,18 +357,6 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
-
- <varlistentry>
- <term><literal>regress_dump_test</literal></term>
- <listitem>
- <para>
- When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
- tests dump and restore of regression database left behind by the
- regression run. Not enabled by default because it is time and resource
- consuming.
- </para>
- </listitem>
- </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index d08eea6693f..cbd9831bf9e 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -268,16 +268,9 @@ else
# should be done in a separate TAP test, but doing it here saves us one full
# regression run.
#
- # This step takes several extra seconds and some extra disk space, so
- # requires an opt-in with the PG_TEST_EXTRA environment variable.
- #
# Do this while the old cluster is running before it is shut down by the
# upgrade test.
- if ( $ENV{PG_TEST_EXTRA}
- && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
- {
- test_regression_dump_restore($oldnode, %node_params);
- }
+ test_regression_dump_restore($oldnode, %node_params);
}
# Initialize a new node for the upgrade.
@@ -590,53 +583,34 @@ sub test_regression_dump_restore
$dst_node->append_conf('postgresql.conf', 'autovacuum = off');
$dst_node->start;
- # Test all formats one by one.
- for my $format ('plain', 'tar', 'directory', 'custom')
- {
- my $dump_file = "$tempdir/regression_dump.$format";
- my $restored_db = 'regression_' . $format;
-
- # Use --create in dump and restore commands so that the restored
- # database has the same configurable variable settings as the original
- # database and the plain dumps taken for comparsion do not differ
- # because of locale changes. Additionally this provides test coverage
- # for --create option.
- $src_node->command_ok(
- [
- 'pg_dump', "-F$format", '--no-sync',
- '-d', $src_node->connstr('regression'),
- '--create', '-f', $dump_file
- ],
- "pg_dump on source instance in $format format");
+ my $dump_file = "$tempdir/regression.dump";
- my @restore_command;
- if ($format eq 'plain')
- {
- # Restore dump in "plain" format using `psql`.
- @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
- }
- else
- {
- @restore_command = [
- 'pg_restore', '--create',
- '-d', 'postgres', $dump_file
- ];
- }
- $dst_node->command_ok(@restore_command,
- "restored dump taken in $format format on destination instance");
+ # Use --create in dump and restore commands so that the restored database
+ # has the same configurable variable settings as the original database so
+ # that the plain dumps taken from both the database taken for comparisong do
+ # not differ because of locale changes. Additionally this provides test
+ # coverage for --create option.
+ #
+ # We use directory format which allows dumping and restoring in parallel to
+ # reduce the test's run time.
+ $src_node->command_ok(
+ [
+ 'pg_dump', '-Fd', '-j2', '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance succeeded");
- my $dst_dump =
- get_dump_for_comparison($dst_node, 'regression',
- 'dest_dump.' . $format, 0);
+ $dst_node->command_ok(
+ [ 'pg_restore', '--create', '-j2', '-d', 'postgres', $dump_file ],
+ "restored dump to destination instance");
- compare_files($src_dump, $dst_dump,
- "dump outputs from original and restored regression database (using $format format) match"
- );
+ my $dst_dump = get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump', 0);
- # Rename the restored database so that it is available for debugging in
- # case the test fails.
- $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
- }
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database match"
+ );
}
# Dump database `db` from the given `node` in plain format and adjust it for
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-24 09:59 ` Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Daniel Gustafsson @ 2025-03-24 09:59 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Michael Paquier <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
> On 24 Mar 2025, at 10:54, Ashutosh Bapat <[email protected]> wrote:
> 0003 - same as 0002 in the previous patch set. It excludes statistics
> from comparison, otherwise the test will fail because of bug reported
> at [1]. Ideally we shouldn't commit this patch so as to test
> statistics dump and restore, but in case we need the test to pass till
> the bug is fixed, we should merge this patch to 0001 before
> committing.
If the reported bug isn't fixed before feature freeze I think we should commit
this regardless as it has clearly shown value by finding bugs (though perhaps
under PG_TEST_EXTRA or in some disconnected till the bug is fixed to limit the
blast-radius in the buildfarm).
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-24 12:14 ` Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-24 12:14 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-24, Ashutosh Bapat wrote:
> One concern I have with directory format is the dumped database is not
> readable. This might make investigating a but identified the test a
> bit more complex.
Oh, it's readable all right. You just need to use `pg_restore -f-` to
read it. No big deal.
So I ran this a few times:
/usr/bin/time make -j8 -Otarget -C /pgsql/build/master check-world -s PROVE_FLAGS="-c -j6" > /dev/null
commenting out the call to test_regression_dump_restore() to test how
much additional runtime does the new test incur.
With test:
136.95user 116.56system 1:13.23elapsed 346%CPU (0avgtext+0avgdata 250704maxresident)k
4928inputs+55333008outputs (114major+14784937minor)pagefaults 0swaps
138.11user 117.43system 1:15.54elapsed 338%CPU (0avgtext+0avgdata 278592maxresident)k
48inputs+55333464outputs (80major+14794494minor)pagefaults 0swaps
137.05user 113.13system 1:08.19elapsed 366%CPU (0avgtext+0avgdata 279272maxresident)k
48inputs+55330064outputs (83major+14758028minor)pagefaults 0swaps
without the new test:
135.46user 114.55system 1:14.69elapsed 334%CPU (0avgtext+0avgdata 145372maxresident)k
32inputs+55155256outputs (105major+14737549minor)pagefaults 0swaps
135.48user 114.57system 1:09.60elapsed 359%CPU (0avgtext+0avgdata 148224maxresident)k
16inputs+55155432outputs (95major+14749502minor)pagefaults 0swaps
133.76user 113.26system 1:14.92elapsed 329%CPU (0avgtext+0avgdata 148064maxresident)k
48inputs+55154952outputs (84major+14749531minor)pagefaults 0swaps
134.06user 113.83system 1:16.09elapsed 325%CPU (0avgtext+0avgdata 145940maxresident)k
32inputs+55155032outputs (83major+14738602minor)pagefaults 0swaps
The increase in duration here is less than a second.
My conclusion with these numbers is that it's not worth hiding this test
in PG_TEST_EXTRA. If we really wanted to save some total test runtime,
it might be better to write a regress schedule file for
027_stream_regress.pl which only takes the test that emit useful WAL,
rather than all tests.
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"The ability of users to misuse tools is, of course, legendary" (David Steele)
https://postgr.es/m/[email protected]
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-25 10:39 ` Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-25 10:39 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Mar 24, 2025 at 5:44 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-24, Ashutosh Bapat wrote:
>
> > One concern I have with directory format is the dumped database is not
> > readable. This might make investigating a but identified the test a
> > bit more complex.
>
> Oh, it's readable all right. You just need to use `pg_restore -f-` to
> read it. No big deal.
>
>
> So I ran this a few times:
> /usr/bin/time make -j8 -Otarget -C /pgsql/build/master check-world -s PROVE_FLAGS="-c -j6" > /dev/null
>
> commenting out the call to test_regression_dump_restore() to test how
> much additional runtime does the new test incur.
>
> With test:
>
> 136.95user 116.56system 1:13.23elapsed 346%CPU (0avgtext+0avgdata 250704maxresident)k
> 4928inputs+55333008outputs (114major+14784937minor)pagefaults 0swaps
>
> 138.11user 117.43system 1:15.54elapsed 338%CPU (0avgtext+0avgdata 278592maxresident)k
> 48inputs+55333464outputs (80major+14794494minor)pagefaults 0swaps
>
> 137.05user 113.13system 1:08.19elapsed 366%CPU (0avgtext+0avgdata 279272maxresident)k
> 48inputs+55330064outputs (83major+14758028minor)pagefaults 0swaps
>
> without the new test:
>
> 135.46user 114.55system 1:14.69elapsed 334%CPU (0avgtext+0avgdata 145372maxresident)k
> 32inputs+55155256outputs (105major+14737549minor)pagefaults 0swaps
>
> 135.48user 114.57system 1:09.60elapsed 359%CPU (0avgtext+0avgdata 148224maxresident)k
> 16inputs+55155432outputs (95major+14749502minor)pagefaults 0swaps
>
> 133.76user 113.26system 1:14.92elapsed 329%CPU (0avgtext+0avgdata 148064maxresident)k
> 48inputs+55154952outputs (84major+14749531minor)pagefaults 0swaps
>
> 134.06user 113.83system 1:16.09elapsed 325%CPU (0avgtext+0avgdata 145940maxresident)k
> 32inputs+55155032outputs (83major+14738602minor)pagefaults 0swaps
>
> The increase in duration here is less than a second.
>
>
> My conclusion with these numbers is that it's not worth hiding this test
> in PG_TEST_EXTRA.
Thanks for the conclusion.
On Mon, Mar 24, 2025 at 3:29 PM Daniel Gustafsson <[email protected]> wrote:
>
> > On 24 Mar 2025, at 10:54, Ashutosh Bapat <[email protected]> wrote:
>
> > 0003 - same as 0002 in the previous patch set. It excludes statistics
> > from comparison, otherwise the test will fail because of bug reported
> > at [1]. Ideally we shouldn't commit this patch so as to test
> > statistics dump and restore, but in case we need the test to pass till
> > the bug is fixed, we should merge this patch to 0001 before
> > committing.
>
> If the reported bug isn't fixed before feature freeze I think we should commit
> this regardless as it has clearly shown value by finding bugs (though perhaps
> under PG_TEST_EXTRA or in some disconnected till the bug is fixed to limit the
> blast-radius in the buildfarm).
Combining Alvaro's and Daniel's recommendations, I think we should
squash all the three of my patches while committing the test if the
bug is not fixed by then. Otherwise we should squash first two patches
and commit it. Just attaching the patches again for reference.
> If we really wanted to save some total test runtime,
> it might be better to write a regress schedule file for
> 027_stream_regress.pl which only takes the test that emit useful WAL,
> rather than all tests.
That's out of scope for this patch, but it seems like an idea worth exploring.
--
Best Wishes,
Ashutosh Bapat
Attachments:
[text/x-patch] 0002-Use-only-one-format-and-make-the-test-run-d-20250324.patch (5.4K, ../../CAExHW5usOKfi-Q1jSi5F50H1mMykAsCayKWEXsES7QKtmwdxtA@mail.gmail.com/2-0002-Use-only-one-format-and-make-the-test-run-d-20250324.patch)
download | inline diff:
From f26f88364a196dc9589ca451cb54f5e514e3422e Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Mon, 24 Mar 2025 11:21:12 +0530
Subject: [PATCH 2/3] Use only one format and make the test run default
According to Alvaro (and I agree with him), the test should be run by
default. Otherwise we get to know about a bug only after buildfarm
animal where it's enabled reports a failure. Further testing only one
format may suffice; since all the formats have shown the same bugs till
now.
If we use --directory format we can use -j which reduces the time taken
by dump/restore test by about 12%.
This patch removes PG_TEST_EXTRA option as well as runs the test only in
directory format with parallelism enabled.
Note for committer: If we decide to accept this change, it should be
merged with the previous commit.
---
doc/src/sgml/regress.sgml | 12 ----
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 76 +++++++++-----------------
2 files changed, 25 insertions(+), 63 deletions(-)
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 237b974b3ab..0e5e8e8f309 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,18 +357,6 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
-
- <varlistentry>
- <term><literal>regress_dump_test</literal></term>
- <listitem>
- <para>
- When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
- tests dump and restore of regression database left behind by the
- regression run. Not enabled by default because it is time and resource
- consuming.
- </para>
- </listitem>
- </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index d08eea6693f..cbd9831bf9e 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -268,16 +268,9 @@ else
# should be done in a separate TAP test, but doing it here saves us one full
# regression run.
#
- # This step takes several extra seconds and some extra disk space, so
- # requires an opt-in with the PG_TEST_EXTRA environment variable.
- #
# Do this while the old cluster is running before it is shut down by the
# upgrade test.
- if ( $ENV{PG_TEST_EXTRA}
- && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
- {
- test_regression_dump_restore($oldnode, %node_params);
- }
+ test_regression_dump_restore($oldnode, %node_params);
}
# Initialize a new node for the upgrade.
@@ -590,53 +583,34 @@ sub test_regression_dump_restore
$dst_node->append_conf('postgresql.conf', 'autovacuum = off');
$dst_node->start;
- # Test all formats one by one.
- for my $format ('plain', 'tar', 'directory', 'custom')
- {
- my $dump_file = "$tempdir/regression_dump.$format";
- my $restored_db = 'regression_' . $format;
-
- # Use --create in dump and restore commands so that the restored
- # database has the same configurable variable settings as the original
- # database and the plain dumps taken for comparsion do not differ
- # because of locale changes. Additionally this provides test coverage
- # for --create option.
- $src_node->command_ok(
- [
- 'pg_dump', "-F$format", '--no-sync',
- '-d', $src_node->connstr('regression'),
- '--create', '-f', $dump_file
- ],
- "pg_dump on source instance in $format format");
+ my $dump_file = "$tempdir/regression.dump";
- my @restore_command;
- if ($format eq 'plain')
- {
- # Restore dump in "plain" format using `psql`.
- @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
- }
- else
- {
- @restore_command = [
- 'pg_restore', '--create',
- '-d', 'postgres', $dump_file
- ];
- }
- $dst_node->command_ok(@restore_command,
- "restored dump taken in $format format on destination instance");
+ # Use --create in dump and restore commands so that the restored database
+ # has the same configurable variable settings as the original database so
+ # that the plain dumps taken from both the database taken for comparisong do
+ # not differ because of locale changes. Additionally this provides test
+ # coverage for --create option.
+ #
+ # We use directory format which allows dumping and restoring in parallel to
+ # reduce the test's run time.
+ $src_node->command_ok(
+ [
+ 'pg_dump', '-Fd', '-j2', '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance succeeded");
- my $dst_dump =
- get_dump_for_comparison($dst_node, 'regression',
- 'dest_dump.' . $format, 0);
+ $dst_node->command_ok(
+ [ 'pg_restore', '--create', '-j2', '-d', 'postgres', $dump_file ],
+ "restored dump to destination instance");
- compare_files($src_dump, $dst_dump,
- "dump outputs from original and restored regression database (using $format format) match"
- );
+ my $dst_dump = get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump', 0);
- # Rename the restored database so that it is available for debugging in
- # case the test fails.
- $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
- }
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database match"
+ );
}
# Dump database `db` from the given `node` in plain format and adjust it for
--
2.34.1
[text/x-patch] 0001-Test-pg_dump-restore-of-regression-objects-20250324.patch (16.5K, ../../CAExHW5usOKfi-Q1jSi5F50H1mMykAsCayKWEXsES7QKtmwdxtA@mail.gmail.com/3-0001-Test-pg_dump-restore-of-regression-objects-20250324.patch)
download | inline diff:
From fcfd0d25ecd374d55970817b4d3ea2aecdd58251 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 27 Jun 2024 10:03:53 +0530
Subject: [PATCH 1/3] Test pg_dump/restore of regression objects
002_pg_upgrade.pl tests pg_upgrade of the regression database left
behind by regression run. Modify it to test dump and restore of the
regression database as well.
Regression database created by regression run contains almost all the
database objects supported by PostgreSQL in various states. Hence the
new testcase covers dump and restore scenarios not covered by individual
dump/restore cases. Till now 002_pg_upgrade only tested dump/restore
through pg_upgrade which only uses binary mode. Many regression tests
mention that they leave objects behind for dump/restore testing but they
are not tested in a non-binary mode. The new testcase closes that
gap.
Testing dump and restore of regression database makes this test run
longer for a relatively smaller benefit. Hence run it only when
explicitly requested by user by specifying "regress_dump_test" in
PG_TEST_EXTRA.
Note For the reviewers:
The new test has uncovered many bugs so far in one year.
1. Introduced by 14e87ffa5c54. Fixed in fd41ba93e4630921a72ed5127cd0d552a8f3f8fc.
2. Introduced by 0413a556990ba628a3de8a0b58be020fd9a14ed0. Reverted in 74563f6b90216180fc13649725179fc119dddeb5.
3. Fixed by d611f8b1587b8f30caa7c0da99ae5d28e914d54f
3. Being discussed on hackers at https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Author: Ashutosh Bapat
Reviewed by: Michael Pacquire, Daniel Gustafsson, Tom Lane, Alvaro Herrera
Discussion: https://www.postgresql.org/message-id/CAExHW5uF5V=Cjecx3_Z=7xfh4rg2Wf61PT+hfquzjBqouRzQJQ@mail.gmail.com
---
doc/src/sgml/regress.sgml | 12 ++
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 144 ++++++++++++++++-
src/test/perl/Makefile | 2 +
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 167 ++++++++++++++++++++
src/test/perl/meson.build | 1 +
5 files changed, 324 insertions(+), 2 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/AdjustDump.pm
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 0e5e8e8f309..237b974b3ab 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,6 +357,18 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>regress_dump_test</literal></term>
+ <listitem>
+ <para>
+ When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
+ tests dump and restore of regression database left behind by the
+ regression run. Not enabled by default because it is time and resource
+ consuming.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 00051b85035..d08eea6693f 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -12,6 +12,7 @@ use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::AdjustUpgrade;
+use PostgreSQL::Test::AdjustDump;
use Test::More;
# Can be changed to test the other modes.
@@ -35,8 +36,8 @@ sub generate_db
"created database with ASCII characters from $from_char to $to_char");
}
-# Filter the contents of a dump before its use in a content comparison.
-# This returns the path to the filtered dump.
+# Filter the contents of a dump before its use in a content comparison for
+# upgrade testing. This returns the path to the filtered dump.
sub filter_dump
{
my ($is_old, $old_version, $dump_file) = @_;
@@ -262,6 +263,21 @@ else
}
}
is($rc, 0, 'regression tests pass');
+
+ # Test dump/restore of the objects left behind by regression. Ideally it
+ # should be done in a separate TAP test, but doing it here saves us one full
+ # regression run.
+ #
+ # This step takes several extra seconds and some extra disk space, so
+ # requires an opt-in with the PG_TEST_EXTRA environment variable.
+ #
+ # Do this while the old cluster is running before it is shut down by the
+ # upgrade test.
+ if ( $ENV{PG_TEST_EXTRA}
+ && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
+ {
+ test_regression_dump_restore($oldnode, %node_params);
+ }
}
# Initialize a new node for the upgrade.
@@ -539,4 +555,128 @@ my $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
compare_files($dump1_filtered, $dump2_filtered,
'old and new dumps match after pg_upgrade');
+# Test dump and restore of objects left behind by the regression run.
+#
+# It is expected that regression tests, which create `regression` database, are
+# run on `src_node`, which in turn, is left in running state. A fresh node is
+# created using given `node_params`, which are expected to be the same ones used
+# to create `src_node`, so as to avoid any differences in the databases.
+#
+# Plain dumps from both the nodes are compared to make sure that all the dumped
+# objects are restored faithfully.
+sub test_regression_dump_restore
+{
+ my ($src_node, %node_params) = @_;
+ my $dst_node = PostgreSQL::Test::Cluster->new('dst_node');
+
+ # Make sure that the source and destination nodes have the same version and
+ # do not use custom install paths. In both the cases, the dump files may
+ # require additional adjustments unknown to code here. Do not run this test
+ # in such a case to avoid utilizing the time and resources unnecessarily.
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default installation");
+ return;
+ }
+
+ # Dump the original database for comparison later.
+ my $src_dump =
+ get_dump_for_comparison($src_node, 'regression', 'src_dump', 1);
+
+ # Setup destination database cluster
+ $dst_node->init(%node_params);
+ # Stabilize stats for comparison.
+ $dst_node->append_conf('postgresql.conf', 'autovacuum = off');
+ $dst_node->start;
+
+ # Test all formats one by one.
+ for my $format ('plain', 'tar', 'directory', 'custom')
+ {
+ my $dump_file = "$tempdir/regression_dump.$format";
+ my $restored_db = 'regression_' . $format;
+
+ # Use --create in dump and restore commands so that the restored
+ # database has the same configurable variable settings as the original
+ # database and the plain dumps taken for comparsion do not differ
+ # because of locale changes. Additionally this provides test coverage
+ # for --create option.
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '--create',
+ '-d', 'postgres', $dump_file
+ ];
+ }
+ $dst_node->command_ok(@restore_command,
+ "restored dump taken in $format format on destination instance");
+
+ my $dst_dump =
+ get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump.' . $format, 0);
+
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database (using $format format) match"
+ );
+
+ # Rename the restored database so that it is available for debugging in
+ # case the test fails.
+ $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
+ }
+}
+
+# Dump database `db` from the given `node` in plain format and adjust it for
+# comparing dumps from the original and the restored database.
+#
+# `file_prefix` is used to create unique names for all dump files so that they
+# remain available for debugging in case the test fails.
+#
+# `adjust_child_columns` is passed to adjust_regress_dumpfile() which actually
+# adjusts the dump output.
+#
+# The name of the file containting adjusted dump is returned.
+sub get_dump_for_comparison
+{
+ my ($node, $db, $file_prefix, $adjust_child_columns) = @_;
+
+ my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
+ my $dump_adjusted = "${dumpfile}_adjusted";
+
+ # Usually we avoid comparing statistics in our tests since it is flaky by
+ # nature. However, if statistics is dumped and restored it is expected to be
+ # restored as it is i.e. the statistics from the original database and that
+ # from the restored database should match. We turn off autovacuum on the
+ # source and the target database to avoid any statistics update during
+ # restore operation. Hence we do not exclude statistics from dump.
+ $node->command_ok(
+ [
+ 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ $dumpfile
+ ],
+ 'dump for comparison succeeded');
+
+ open(my $dh, '>', $dump_adjusted)
+ || die
+ "could not open $dump_adjusted for writing the adjusted dump: $!";
+ print $dh adjust_regress_dumpfile(slurp_file($dumpfile),
+ $adjust_child_columns);
+ close($dh);
+
+ return $dump_adjusted;
+}
+
done_testing();
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index d82fb67540e..def89650ead 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustUpgrade.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustDump.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
uninstall:
@@ -36,6 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
endif
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..74b9a60cf34
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,167 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustDump - helper module for dump and restore tests
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::AdjustDump;
+
+ # Adjust contents of dump output file so that dump output from original
+ # regression database and that from the restored regression database match
+ $dump = adjust_regress_dumpfile($dump, $adjust_child_columns);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustDump> encapsulates various hacks needed to
+compare the results of dump and restore tests
+
+=cut
+
+package PostgreSQL::Test::AdjustDump;
+
+use strict;
+use warnings FATAL => 'all';
+
+use Exporter 'import';
+use Test::More;
+
+our @EXPORT = qw(
+ adjust_regress_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $dump = adjust_regress_dumpfile($dump, $adjust_child_columns)
+
+If we take dump of the regression database left behind after running regression
+tests, restore the dump, and take dump of the restored regression database, the
+outputs of both the dumps differ in the following cases. This routine adjusts
+the given dump so that dump outputs from the original and restored database,
+respectively, match.
+
+Case 1: Some regression tests purposefully create child tables in such a way
+that the order of their inherited columns differ from column orders of their
+respective parents. In the restored database, however, the order of their
+inherited columns are same as that of their respective parents. Thus the column
+orders of these child tables in the original database and those in the restored
+database differ, causing difference in the dump outputs. See MergeAttributes()
+and dumpTableSchema() for details. This routine rearranges the column
+declarations in the relevant C<CREATE TABLE... INHERITS> statements in the dump
+file from original database to match those from the restored database. We could,
+instead, adjust the statements in the dump from the restored database to match
+those from original database or adjust both to a canonical order. But we have
+chosen to adjust the statements in the dump from original database for no
+particular reason.
+
+Case 2: When dumping COPY statements the columns are ordered by their attribute
+number by fmtCopyColumnList(). If a column is added to a parent table after a
+child has inherited the parent and the child has its own columns, the attribute
+number of the column changes after restoring the child table. This is because
+when executing the dumped C<CREATE TABLE... INHERITS> statement all the parent
+attributes are created before any child attributes. Thus the order of columns in
+COPY statements dumped from the original and the restored databases,
+respectively, differs. Such tables in regression tests are listed below. It is
+hard to adjust the column order in the COPY statement along with the data. Hence
+we just remove such COPY statements from the dump output.
+
+Additionally the routine adjusts blank and new lines to avoid noise.
+
+Note: Usually we avoid comparing statistics in our tests since it is flaky by
+nature. However, if statistics is dumped and restored it is expected to be
+restored as it is i.e. the statistics from the original database and that from
+the restored database should match. Hence we do not filter statistics from dump,
+if it's dumped.
+
+Arguments:
+
+=over
+
+=item C<dump>: Contents of dump file
+
+=item C<adjust_child_columns>: 1 indicates that the given dump file requires
+adjusting columns in the child tables; usually when the dump is from original
+database. 0 indicates no such adjustment is needed; usually when the dump is
+from restored database.
+
+=back
+
+Returns the adjusted dump text.
+
+=cut
+
+sub adjust_regress_dumpfile
+{
+ my ($dump, $adjust_child_columns) = @_;
+
+ # use Unix newlines
+ $dump =~ s/\r\n/\n/g;
+
+ # Adjust the CREATE TABLE ... INHERITS statements.
+ if ($adjust_child_columns)
+ {
+ my $saved_dump = $dump;
+
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_stored_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_stored_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_virtual_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_virtual_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c1\s\()
+ (\n\s+int_four\sbigint),
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint)/$1$4,$2,$3/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c1 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c2\s\()
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint),
+ (\n\s+int_four\sbigint)/$1$3,$4,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c2 adjustments');
+ }
+
+ # Remove COPY statements with differing column order
+ for my $table (
+ 'public\.b_star', 'public\.c_star',
+ 'public\.cc2', 'public\.d_star',
+ 'public\.e_star', 'public\.f_star',
+ 'public\.renamecolumnanother', 'public\.renamecolumnchild',
+ 'public\.test_type_diff2_c1', 'public\.test_type_diff2_c2',
+ 'public\.test_type_diff_c')
+ {
+ $dump =~ s/^COPY\s$table\s\(.+?^\\\.$//sm;
+ }
+
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
+ return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff --git a/src/test/perl/meson.build b/src/test/perl/meson.build
index 58e30f15f9d..492ca571ff8 100644
--- a/src/test/perl/meson.build
+++ b/src/test/perl/meson.build
@@ -14,4 +14,5 @@ install_data(
'PostgreSQL/Test/Cluster.pm',
'PostgreSQL/Test/BackgroundPsql.pm',
'PostgreSQL/Test/AdjustUpgrade.pm',
+ 'PostgreSQL/Test/AdjustDump.pm',
install_dir: dir_pgxs / 'src/test/perl/PostgreSQL/Test')
base-commit: 73eba5004a06a744b6b8570e42432b9e9f75997b
--
2.34.1
[text/x-patch] 0003-Do-not-dump-statistics-in-the-file-dumped-f-20250324.patch (2.1K, ../../CAExHW5usOKfi-Q1jSi5F50H1mMykAsCayKWEXsES7QKtmwdxtA@mail.gmail.com/4-0003-Do-not-dump-statistics-in-the-file-dumped-f-20250324.patch)
download | inline diff:
From 435c659489b34a803675abb65144fab6f0550432 Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Tue, 25 Feb 2025 11:42:51 +0530
Subject: [PATCH 3/3] Do not dump statistics in the file dumped for comparison
The dumped and restored statistics of a materialized view may differ as
reported in [1]. Hence do not dump the statistics to avoid differences
in the dump output from the original and restored database.
[1] https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Ashutosh Bapat
---
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index cbd9831bf9e..abe93a49258 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -630,15 +630,15 @@ sub get_dump_for_comparison
my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
my $dump_adjusted = "${dumpfile}_adjusted";
- # Usually we avoid comparing statistics in our tests since it is flaky by
- # nature. However, if statistics is dumped and restored it is expected to be
- # restored as it is i.e. the statistics from the original database and that
- # from the restored database should match. We turn off autovacuum on the
- # source and the target database to avoid any statistics update during
- # restore operation. Hence we do not exclude statistics from dump.
+ # If statistics is dumped and restored it is expected to be restored as it
+ # is i.e. the statistics from the original database and that from the
+ # restored database should match. We turn off autovacuum on the source and
+ # the target database to avoid any statistics update during restore
+ # operation. But as of now, there are cases when statistics is not being
+ # restored faithfully. Hence for now do not dump statistics.
$node->command_ok(
[
- 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ 'pg_dump', '--no-sync', '--no-statistics', '-d', $node->connstr($db), '-f',
$dumpfile
],
'dump for comparison succeeded');
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-27 12:31 ` vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: vignesh C @ 2025-03-27 12:31 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 25 Mar 2025 at 16:09, Ashutosh Bapat
<[email protected]> wrote:
>
> On Mon, Mar 24, 2025 at 5:44 PM Alvaro Herrera <[email protected]> wrote:
> >
> > On 2025-Mar-24, Ashutosh Bapat wrote:
> >
> > > One concern I have with directory format is the dumped database is not
> > > readable. This might make investigating a but identified the test a
> > > bit more complex.
> >
> > Oh, it's readable all right. You just need to use `pg_restore -f-` to
> > read it. No big deal.
> >
> >
> > So I ran this a few times:
> > /usr/bin/time make -j8 -Otarget -C /pgsql/build/master check-world -s PROVE_FLAGS="-c -j6" > /dev/null
> >
> > commenting out the call to test_regression_dump_restore() to test how
> > much additional runtime does the new test incur.
> >
> > With test:
> >
> > 136.95user 116.56system 1:13.23elapsed 346%CPU (0avgtext+0avgdata 250704maxresident)k
> > 4928inputs+55333008outputs (114major+14784937minor)pagefaults 0swaps
> >
> > 138.11user 117.43system 1:15.54elapsed 338%CPU (0avgtext+0avgdata 278592maxresident)k
> > 48inputs+55333464outputs (80major+14794494minor)pagefaults 0swaps
> >
> > 137.05user 113.13system 1:08.19elapsed 366%CPU (0avgtext+0avgdata 279272maxresident)k
> > 48inputs+55330064outputs (83major+14758028minor)pagefaults 0swaps
> >
> > without the new test:
> >
> > 135.46user 114.55system 1:14.69elapsed 334%CPU (0avgtext+0avgdata 145372maxresident)k
> > 32inputs+55155256outputs (105major+14737549minor)pagefaults 0swaps
> >
> > 135.48user 114.57system 1:09.60elapsed 359%CPU (0avgtext+0avgdata 148224maxresident)k
> > 16inputs+55155432outputs (95major+14749502minor)pagefaults 0swaps
> >
> > 133.76user 113.26system 1:14.92elapsed 329%CPU (0avgtext+0avgdata 148064maxresident)k
> > 48inputs+55154952outputs (84major+14749531minor)pagefaults 0swaps
> >
> > 134.06user 113.83system 1:16.09elapsed 325%CPU (0avgtext+0avgdata 145940maxresident)k
> > 32inputs+55155032outputs (83major+14738602minor)pagefaults 0swaps
> >
> > The increase in duration here is less than a second.
> >
> >
> > My conclusion with these numbers is that it's not worth hiding this test
> > in PG_TEST_EXTRA.
>
> Thanks for the conclusion.
>
> On Mon, Mar 24, 2025 at 3:29 PM Daniel Gustafsson <[email protected]> wrote:
> >
> > > On 24 Mar 2025, at 10:54, Ashutosh Bapat <[email protected]> wrote:
> >
> > > 0003 - same as 0002 in the previous patch set. It excludes statistics
> > > from comparison, otherwise the test will fail because of bug reported
> > > at [1]. Ideally we shouldn't commit this patch so as to test
> > > statistics dump and restore, but in case we need the test to pass till
> > > the bug is fixed, we should merge this patch to 0001 before
> > > committing.
> >
> > If the reported bug isn't fixed before feature freeze I think we should commit
> > this regardless as it has clearly shown value by finding bugs (though perhaps
> > under PG_TEST_EXTRA or in some disconnected till the bug is fixed to limit the
> > blast-radius in the buildfarm).
>
> Combining Alvaro's and Daniel's recommendations, I think we should
> squash all the three of my patches while committing the test if the
> bug is not fixed by then. Otherwise we should squash first two patches
> and commit it. Just attaching the patches again for reference.
Couple of minor thoughts:
1) I felt this error message is not conveying the error message correctly:
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default
installation");
+ return;
+ }
how about something like below:
fail("source and destination nodes must have the same PostgreSQL
version and default installation paths");
2) Should "`" be ' or " here, we generally use "`" to enclose commands:
+# It is expected that regression tests, which create `regression` database, are
+# run on `src_node`, which in turn, is left in running state. A fresh node is
+# created using given `node_params`, which are expected to be the
same ones used
+# to create `src_node`, so as to avoid any differences in the databases.
There are few other instances similarly in the file.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
@ 2025-03-27 16:31 ` Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-27 16:31 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 27, 2025 at 6:01 PM vignesh C <[email protected]> wrote:
>
> On Tue, 25 Mar 2025 at 16:09, Ashutosh Bapat
> <[email protected]> wrote:
> >
> > On Mon, Mar 24, 2025 at 5:44 PM Alvaro Herrera <[email protected]> wrote:
> > >
> > > On 2025-Mar-24, Ashutosh Bapat wrote:
> > >
> > > > One concern I have with directory format is the dumped database is not
> > > > readable. This might make investigating a but identified the test a
> > > > bit more complex.
> > >
> > > Oh, it's readable all right. You just need to use `pg_restore -f-` to
> > > read it. No big deal.
> > >
> > >
> > > So I ran this a few times:
> > > /usr/bin/time make -j8 -Otarget -C /pgsql/build/master check-world -s PROVE_FLAGS="-c -j6" > /dev/null
> > >
> > > commenting out the call to test_regression_dump_restore() to test how
> > > much additional runtime does the new test incur.
> > >
> > > With test:
> > >
> > > 136.95user 116.56system 1:13.23elapsed 346%CPU (0avgtext+0avgdata 250704maxresident)k
> > > 4928inputs+55333008outputs (114major+14784937minor)pagefaults 0swaps
> > >
> > > 138.11user 117.43system 1:15.54elapsed 338%CPU (0avgtext+0avgdata 278592maxresident)k
> > > 48inputs+55333464outputs (80major+14794494minor)pagefaults 0swaps
> > >
> > > 137.05user 113.13system 1:08.19elapsed 366%CPU (0avgtext+0avgdata 279272maxresident)k
> > > 48inputs+55330064outputs (83major+14758028minor)pagefaults 0swaps
> > >
> > > without the new test:
> > >
> > > 135.46user 114.55system 1:14.69elapsed 334%CPU (0avgtext+0avgdata 145372maxresident)k
> > > 32inputs+55155256outputs (105major+14737549minor)pagefaults 0swaps
> > >
> > > 135.48user 114.57system 1:09.60elapsed 359%CPU (0avgtext+0avgdata 148224maxresident)k
> > > 16inputs+55155432outputs (95major+14749502minor)pagefaults 0swaps
> > >
> > > 133.76user 113.26system 1:14.92elapsed 329%CPU (0avgtext+0avgdata 148064maxresident)k
> > > 48inputs+55154952outputs (84major+14749531minor)pagefaults 0swaps
> > >
> > > 134.06user 113.83system 1:16.09elapsed 325%CPU (0avgtext+0avgdata 145940maxresident)k
> > > 32inputs+55155032outputs (83major+14738602minor)pagefaults 0swaps
> > >
> > > The increase in duration here is less than a second.
> > >
> > >
> > > My conclusion with these numbers is that it's not worth hiding this test
> > > in PG_TEST_EXTRA.
> >
> > Thanks for the conclusion.
> >
> > On Mon, Mar 24, 2025 at 3:29 PM Daniel Gustafsson <[email protected]> wrote:
> > >
> > > > On 24 Mar 2025, at 10:54, Ashutosh Bapat <[email protected]> wrote:
> > >
> > > > 0003 - same as 0002 in the previous patch set. It excludes statistics
> > > > from comparison, otherwise the test will fail because of bug reported
> > > > at [1]. Ideally we shouldn't commit this patch so as to test
> > > > statistics dump and restore, but in case we need the test to pass till
> > > > the bug is fixed, we should merge this patch to 0001 before
> > > > committing.
> > >
> > > If the reported bug isn't fixed before feature freeze I think we should commit
> > > this regardless as it has clearly shown value by finding bugs (though perhaps
> > > under PG_TEST_EXTRA or in some disconnected till the bug is fixed to limit the
> > > blast-radius in the buildfarm).
> >
> > Combining Alvaro's and Daniel's recommendations, I think we should
> > squash all the three of my patches while committing the test if the
> > bug is not fixed by then. Otherwise we should squash first two patches
> > and commit it. Just attaching the patches again for reference.
>
> Couple of minor thoughts:
> 1) I felt this error message is not conveying the error message correctly:
> + if ($src_node->pg_version != $dst_node->pg_version
> + or defined $src_node->{_install_path})
> + {
> + fail("same version dump and restore test using default
> installation");
> + return;
> + }
>
> how about something like below:
> fail("source and destination nodes must have the same PostgreSQL
> version and default installation paths");
The text in ok(), fail() etc. are test names and not error messages.
See [1]. Your suggestion and other versions that I came up with became
too verbose to be test names. So I think the text here is compromise
between conveying enough information and not being too long. We
usually have to pick the testname and lookup the test code to
investigate the failure. This text serves that purpose.
>
> 2) Should "`" be ' or " here, we generally use "`" to enclose commands:
> +# It is expected that regression tests, which create `regression` database, are
> +# run on `src_node`, which in turn, is left in running state. A fresh node is
> +# created using given `node_params`, which are expected to be the
> same ones used
> +# to create `src_node`, so as to avoid any differences in the databases.
Looking at prologues or some other functions, I see that we don't add
any decoration around the name of the argument. Hence dropped ``
altogether. Will post it with the next set of patches.
[1] https://metacpan.org/pod/Test::More
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-27 17:15 ` Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:32 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-27 17:15 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-27, Ashutosh Bapat wrote:
> On Thu, Mar 27, 2025 at 6:01 PM vignesh C <[email protected]> wrote:
> > Couple of minor thoughts:
> > 1) I felt this error message is not conveying the error message correctly:
> > + if ($src_node->pg_version != $dst_node->pg_version
> > + or defined $src_node->{_install_path})
> > + {
> > + fail("same version dump and restore test using default
> > installation");
> > + return;
> > + }
> >
> > how about something like below:
> > fail("source and destination nodes must have the same PostgreSQL
> > version and default installation paths");
>
> The text in ok(), fail() etc. are test names and not error messages.
> See [1]. Your suggestion and other versions that I came up with became
> too verbose to be test names. So I think the text here is compromise
> between conveying enough information and not being too long. We
> usually have to pick the testname and lookup the test code to
> investigate the failure. This text serves that purpose.
Maybe
fail("roundtrip dump/restore of the regression database")
BTW another idea to shorten this tests's runtime might be to try and
identify which of parallel_schedule tests leave objects behind and
create a shorter schedule with only those (a possible implementation
might keep a list of the slow tests that don't leave any useful object
behind, then filter parallel_schedule to exclude those; this ensures
test files created in the future are still used.)
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"I love the Postgres community. It's all about doing things _properly_. :-)"
(David Garamond)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-28 01:36 ` Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: Michael Paquier @ 2025-03-28 01:36 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 27, 2025 at 06:15:06PM +0100, Alvaro Herrera wrote:
> BTW another idea to shorten this tests's runtime might be to try and
> identify which of parallel_schedule tests leave objects behind and
> create a shorter schedule with only those (a possible implementation
> might keep a list of the slow tests that don't leave any useful object
> behind, then filter parallel_schedule to exclude those; this ensures
> test files created in the future are still used.)
I'm not much a fan of approaches that require an extra schedule,
because this is prone to forget the addition of objects that we'd want
to cover for the scope of this thread with the dump/restore
inter-dependencies, failing our goal of having more coverage. And
history has proven that we are quite bad at maintaining multiple
schedules for the regression test suite (remember the serial one or
the standby one in pg_regress?). So we should really do things so as
the schedules are down to a strict minimum: 1.
If we're worried about the time taken by the test (spoiler: I am and
the upgrade tests already show always as last to finish in parallel
runs), I would recommend to put that under a PG_TEST_EXTRA. I'm OK to
add the switch to my buildfarm animals if this option is the consensus
and if it gets into the tree.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
@ 2025-03-28 06:50 ` Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-28 06:50 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 28, 2025 at 7:07 AM Michael Paquier <[email protected]> wrote:
>
> On Thu, Mar 27, 2025 at 06:15:06PM +0100, Alvaro Herrera wrote:
> > BTW another idea to shorten this tests's runtime might be to try and
> > identify which of parallel_schedule tests leave objects behind and
> > create a shorter schedule with only those (a possible implementation
> > might keep a list of the slow tests that don't leave any useful object
> > behind, then filter parallel_schedule to exclude those; this ensures
> > test files created in the future are still used.)
>
> I'm not much a fan of approaches that require an extra schedule,
> because this is prone to forget the addition of objects that we'd want
> to cover for the scope of this thread with the dump/restore
> inter-dependencies, failing our goal of having more coverage. And
> history has proven that we are quite bad at maintaining multiple
> schedules for the regression test suite (remember the serial one or
> the standby one in pg_regress?). So we should really do things so as
> the schedules are down to a strict minimum: 1.
I see Alvaro's point about using a different and minimal schedule. We
already have 002_pg_upgrade and 027_stream_ as candidates which could
use schedules other than default and avoid wasting CPU cycles.
But I also agree with your opinion that maintaining multiple schedules
is painful and prone to errors.
What we could do is to create the schedule files automatically during
build. The automation script will require to know which file to place
in which schedules. That information could be either part of the sql
file itself or could be in a separate text file. For example, every
SQL file has the following line listing all the schedules that this
SQL file should be part of. E.g.
-- schedules: parallel, serial, upgrade
The automated script looks at every .sql file in a given sql directory
and creates the schedule files containing all the SQL files which had
respective schedules mentioned in their "schedule" annotation. The
automation script would flag SQL files that do not have scheduled
annotation so any new file added won't be missed. However, we will
still miss a SQL file if it wasn't part of a given schedule and later
acquired some changes which required it to be added to a new schedule.
If we go this route, we could make 'make check-tests' better. We could
add another annotation for depends listing all the SQL files that a
given SQL file depends upon. make check-tests would collect all
dependencies, sort them and run all the dependencies as well.
Of course that's out of scope for this patch. We don't have time left
for this in PG 18.
>
> If we're worried about the time taken by the test (spoiler: I am and
> the upgrade tests already show always as last to finish in parallel
> runs), I would recommend to put that under a PG_TEST_EXTRA. I'm OK to
> add the switch to my buildfarm animals if this option is the consensus
> and if it gets into the tree.
I would prefer to run this test by default as Alvaro mentioned
previously. But if that means that we won't get this test committed at
all, I am ok putting it under PG_TEST_EXTRA. (Hence I have kept 0001
and 0002 separate.) But I will be disappointed if the test, which has
unearthed four bugs in a year alone, does not get committed to PG 18
because of this debate.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-28 12:27 ` Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-28 12:27 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 28, 2025 at 12:20 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Fri, Mar 28, 2025 at 7:07 AM Michael Paquier <[email protected]> wrote:
> >
> > On Thu, Mar 27, 2025 at 06:15:06PM +0100, Alvaro Herrera wrote:
> > > BTW another idea to shorten this tests's runtime might be to try and
> > > identify which of parallel_schedule tests leave objects behind and
> > > create a shorter schedule with only those (a possible implementation
> > > might keep a list of the slow tests that don't leave any useful object
> > > behind, then filter parallel_schedule to exclude those; this ensures
> > > test files created in the future are still used.)
> >
> > I'm not much a fan of approaches that require an extra schedule,
> > because this is prone to forget the addition of objects that we'd want
> > to cover for the scope of this thread with the dump/restore
> > inter-dependencies, failing our goal of having more coverage. And
> > history has proven that we are quite bad at maintaining multiple
> > schedules for the regression test suite (remember the serial one or
> > the standby one in pg_regress?). So we should really do things so as
> > the schedules are down to a strict minimum: 1.
>
> I see Alvaro's point about using a different and minimal schedule. We
> already have 002_pg_upgrade and 027_stream_ as candidates which could
> use schedules other than default and avoid wasting CPU cycles.
> But I also agree with your opinion that maintaining multiple schedules
> is painful and prone to errors.
>
> What we could do is to create the schedule files automatically during
> build. The automation script will require to know which file to place
> in which schedules. That information could be either part of the sql
> file itself or could be in a separate text file. For example, every
> SQL file has the following line listing all the schedules that this
> SQL file should be part of. E.g.
>
> -- schedules: parallel, serial, upgrade
>
> The automated script looks at every .sql file in a given sql directory
> and creates the schedule files containing all the SQL files which had
> respective schedules mentioned in their "schedule" annotation. The
> automation script would flag SQL files that do not have scheduled
> annotation so any new file added won't be missed. However, we will
> still miss a SQL file if it wasn't part of a given schedule and later
> acquired some changes which required it to be added to a new schedule.
>
> If we go this route, we could make 'make check-tests' better. We could
> add another annotation for depends listing all the SQL files that a
> given SQL file depends upon. make check-tests would collect all
> dependencies, sort them and run all the dependencies as well.
>
> Of course that's out of scope for this patch. We don't have time left
> for this in PG 18.
I spent several hours today examining each SQL file to decide whether
or not it has "interesting" objects that it leaves behind for
dump/restore test. I came up with attached schedule - which may not be
accurate since I it would require much more time to examine all tests
to get an accurate schedule. But what I have got may be close enough.
With that we could save about 6 seconds on my laptop. If we further
compact the schedule reorganizing the parallel groups we may shave
some more seconds.
no modifications to parallel schedule
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.84s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.80s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
41.37s 28 subtests passed
with attached modified parallel schedule
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
36.13s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
35.86s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
36.33s 28 subtests passed
1/1 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK
36.02s 28 subtests passed
However, it's a very painful process to come up with the schedule and
more painful and error prone to maintain it. It could take many days
to come up with the right schedule which can become inaccurate the
moment next SQL file is added OR an existing file is modified to
add/drop "interesting" objects.
--
Best Wishes,
Ashutosh Bapat
Attachments:
[application/octet-stream] parallel_schedule_dump_restore (4.6K, ../../CAExHW5teUDXYR+DyoTP=NJw_=gUy1g=bCmVbKP5+UhRW=Nm0qw@mail.gmail.com/2-parallel_schedule_dump_restore)
download
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-28 14:11 ` Alvaro Herrera <[email protected]>
2025-03-28 14:22 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-28 14:11 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Michael Paquier <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-28, Ashutosh Bapat wrote:
> However, it's a very painful process to come up with the schedule and
> more painful and error prone to maintain it. It could take many days
> to come up with the right schedule which can become inaccurate the
> moment next SQL file is added OR an existing file is modified to
> add/drop "interesting" objects.
Hmm, I didn't mean that we'd maintain a separate schedule. I meant that
we'd take the existing schedule, then apply some Perl magic to it that
grep-outs the tests that we know to contribute nothing, and generate a
new schedule file dynamically. We don't need to maintain a separate
schedule file.
You're right that if an existing uninteresting test is modified to
create interesting objects, we'd lose coverage of those objects. That
seems a much smaller problem to me. So it's just a matter of doing some
Perl map/grep to generate a new schedule file using the attached
exclusion file.
(For what it's worth, what I did to try to determine which tests to
include, rather than scan each file manually, is to run pg_regress with
"test_setup thetest tablespace", then dump the regression database, and
see if anything is there that's not in the dump when I just with just
"test_setup tablespace". I didn't carry the experiment to completion
though.)
For the future, we could annotate each test as you said, either by
adding a marker on the test file itself, or by adding something next to
its name in the schedule file, so the schedule file could look like:
test: plancache(dump_ignore) limit(stream_ignore) plpgsql copy2
temp(stream_ignore,dump_ignore) domain rangefuncs(stream_ignore)
prepare conversion truncate alter_table
sequence polymorphism rowtypes returning largeobject with xml
... and so on.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
# This file lists tests to skip on pg_upgrade/t/002_pg_upgrade.pl
advisory_lock
amutils
async
bitmapops
char
combocid
comments
copy
copy2
copydml
copyencoding
copyselect
database
dbsize
delete
drop_if_exists
drop_operator
equivclass
errors
explain
expressions
functional_deps
groupingsets
guc
hash_func
horology
incremental_sort
infinite_recurse
join_hash
json_encoding
jsonb_jsonpath
jsonpath
jsonpath_encoding
lock
md5
memoize
merge
misc_sanity
mvcc
oidjoins
opr_sanity
partition_aggregate
partition_join
partition_prune
plancache
portals
portals_p2
predicate
prepare
prepared_xacts
psql
psql_crosstab
psql_pipeline
regex
regproc
returning
sanity_check
select
select_distinct
select_distinct_on
select_having
select_implicit
select_parallel
stats_import
strings
subselect
sysviews
tablesample
temp
text
tid
tidrangescan
tidscan
transactions
truncate
tsrf
tstypes
txid
type_sanity
unicode
union
update
vacuum
vacuum_parallel
window
xmlmap
Attachments:
[text/plain] dump_roundtrip_exclude (940B, ../../[email protected]/2-dump_roundtrip_exclude)
download | inline:
# This file lists tests to skip on pg_upgrade/t/002_pg_upgrade.pl
advisory_lock
amutils
async
bitmapops
char
combocid
comments
copy
copy2
copydml
copyencoding
copyselect
database
dbsize
delete
drop_if_exists
drop_operator
equivclass
errors
explain
expressions
functional_deps
groupingsets
guc
hash_func
horology
incremental_sort
infinite_recurse
join_hash
json_encoding
jsonb_jsonpath
jsonpath
jsonpath_encoding
lock
md5
memoize
merge
misc_sanity
mvcc
oidjoins
opr_sanity
partition_aggregate
partition_join
partition_prune
plancache
portals
portals_p2
predicate
prepare
prepared_xacts
psql
psql_crosstab
psql_pipeline
regex
regproc
returning
sanity_check
select
select_distinct
select_distinct_on
select_having
select_implicit
select_parallel
stats_import
strings
subselect
sysviews
tablesample
temp
text
tid
tidrangescan
tidscan
transactions
truncate
tsrf
tstypes
txid
type_sanity
unicode
union
update
vacuum
vacuum_parallel
window
xmlmap
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-28 14:22 ` Tom Lane <[email protected]>
2025-03-28 18:12 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Tom Lane @ 2025-03-28 14:22 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Alvaro Herrera <[email protected]> writes:
> Hmm, I didn't mean that we'd maintain a separate schedule. I meant that
> we'd take the existing schedule, then apply some Perl magic to it that
> grep-outs the tests that we know to contribute nothing, and generate a
> new schedule file dynamically. We don't need to maintain a separate
> schedule file.
This seems like a fundamentally broken approach to me.
The entire argument for using the core regression tests as a source of
data to test dump/restore is that, more or less "for free", we can
expect to get coverage when new SQL language features are added.
That's always been a little bit questionable --- there's a temptation
to drop objects again at the end of a test script. But with this,
it becomes a complete crapshoot whether the objects you need will be
included in the dump.
I think instead of going this direction, we really need to create a
separately-purposed script that simply creates "one of everything"
without doing anything else (except maybe loading a little data).
I believe it'd be a lot easier to remember to add to that when
inventing new SQL than to remember to leave something behind from the
core regression tests. This would also be far faster to run than any
approach that involves picking a random subset of the core test
scripts.
regards, tom lane
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 14:22 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
@ 2025-03-28 18:12 ` Alvaro Herrera <[email protected]>
2025-03-31 11:37 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-31 21:17 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-28 18:12 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-28, Tom Lane wrote:
> I think instead of going this direction, we really need to create a
> separately-purposed script that simply creates "one of everything"
> without doing anything else (except maybe loading a little data).
> I believe it'd be a lot easier to remember to add to that when
> inventing new SQL than to remember to leave something behind from the
> core regression tests. This would also be far faster to run than any
> approach that involves picking a random subset of the core test
> scripts.
FWIW this sounds closely related to what I tried to do with
src/test/modules/test_ddl_deparse; it's currently incomplete, but maybe
we can use that as a starting point.
--
Álvaro Herrera Breisgau, Deutschland — https://www.EnterpriseDB.com/
"Always assume the user will do much worse than the stupidest thing
you can imagine." (Julien PUYDT)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 14:22 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-03-28 18:12 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-31 11:37 ` Ashutosh Bapat <[email protected]>
2025-03-31 11:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-31 11:37 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tom Lane <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 28, 2025 at 11:43 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-28, Tom Lane wrote:
>
> > I think instead of going this direction, we really need to create a
> > separately-purposed script that simply creates "one of everything"
> > without doing anything else (except maybe loading a little data).
> > I believe it'd be a lot easier to remember to add to that when
> > inventing new SQL than to remember to leave something behind from the
> > core regression tests. This would also be far faster to run than any
> > approach that involves picking a random subset of the core test
> > scripts.
It's easier to remember to do something or not do something in the
same file than in some other file. I find it hard to believe that
introducing another set of SQL files somewhere far from regress would
make this problem easier.
The number of states in which objects can be left behind in the
regress/sql is very large - and maintaining that 1:1 in some other set
of scripts is impossible unless it's automated.
> FWIW this sounds closely related to what I tried to do with
> src/test/modules/test_ddl_deparse; it's currently incomplete, but maybe
> we can use that as a starting point.
create_table.sql in test_ddl_deparse has only one statement creating
an inheritance table whereas there are dozens of different states of
parent/child tables created by regress. It will require a lot of work
to bridge the gap between regress_ddl_deparse and regress and more
work to maintain it.
I might be missing something in your ideas.
IMO, whatever we do it should rely on a single set of files. One
possible way could be to break the existing files into three files
each, containing DDL, DML and queries from those files respectively
and create three schedules DDL, DML and queries containing the
respective files. These schedules will be run as required. Standard
regression run runs all the three schedules one by one. But
002_pg_upgrade will run DDL and DML on the source database and run
queries on target - thus checking sanity of the dump/restore or
pg_upgrade beyond just the dump comparison. 027_stream_regress might
run DDL, DML on the source server and queries on the target.
But that too is easier said than done for:
1. Our tests mix all three kinds of statements and also rely on the
order in which they are run. It will require some significant effort
to carefully separate the statements.
2. With the new set of files backpatching would become hard.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 14:22 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-03-28 18:12 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-31 11:37 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-31 11:54 ` Ashutosh Bapat <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-31 11:54 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tom Lane <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Daniel Gustafsson <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Mar 31, 2025 at 5:07 PM Ashutosh Bapat
<[email protected]> wrote:
>
The bug related to materialized views has been fixed and now the test
passes even if we compare statistics from dumped and restored
databases. Hence removing 0003. In the attached patchset I have also
addressed Vignesh's below comment
On Thu, Mar 27, 2025 at 10:01 PM Ashutosh Bapat
<[email protected]> wrote:
>
> On Thu, Mar 27, 2025 at 6:01 PM vignesh C <[email protected]> wrote:
> >
> > 2) Should "`" be ' or " here, we generally use "`" to enclose commands:
> > +# It is expected that regression tests, which create `regression` database, are
> > +# run on `src_node`, which in turn, is left in running state. A fresh node is
> > +# created using given `node_params`, which are expected to be the
> > same ones used
> > +# to create `src_node`, so as to avoid any differences in the databases.
>
> Looking at prologues or some other functions, I see that we don't add
> any decoration around the name of the argument. Hence dropped ``
> altogether. Will post it with the next set of patches.
--
Best Wishes,
Ashutosh Bapat
Attachments:
[text/x-patch] 0002-Use-only-one-format-and-make-the-test-run-d-20250331.patch (5.4K, ../../CAExHW5vVFtCejh+UYzNxMGSXOfJ_1xwi5aQHQfemqJgFmkyK5Q@mail.gmail.com/2-0002-Use-only-one-format-and-make-the-test-run-d-20250331.patch)
download | inline diff:
From 5ef4a15bf229d104028eac3a046636453e1e05fc Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Mon, 24 Mar 2025 11:21:12 +0530
Subject: [PATCH 2/2] Use only one format and make the test run default
According to Alvaro (and I agree with him), the test should be run by
default. Otherwise we get to know about a bug only after buildfarm
animal where it's enabled reports a failure. Further testing only one
format may suffice; since all the formats have shown the same bugs till
now.
If we use --directory format we can use -j which reduces the time taken
by dump/restore test by about 12%.
This patch removes PG_TEST_EXTRA option as well as runs the test only in
directory format with parallelism enabled.
Note for committer: If we decide to accept this change, it should be
merged with the previous commit.
---
doc/src/sgml/regress.sgml | 12 ----
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 76 +++++++++-----------------
2 files changed, 25 insertions(+), 63 deletions(-)
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 237b974b3ab..0e5e8e8f309 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,18 +357,6 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
-
- <varlistentry>
- <term><literal>regress_dump_test</literal></term>
- <listitem>
- <para>
- When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
- tests dump and restore of regression database left behind by the
- regression run. Not enabled by default because it is time and resource
- consuming.
- </para>
- </listitem>
- </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 8d22d538529..f7d5b96ecd2 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -268,16 +268,9 @@ else
# should be done in a separate TAP test, but doing it here saves us one full
# regression run.
#
- # This step takes several extra seconds and some extra disk space, so
- # requires an opt-in with the PG_TEST_EXTRA environment variable.
- #
# Do this while the old cluster is running before it is shut down by the
# upgrade test.
- if ( $ENV{PG_TEST_EXTRA}
- && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
- {
- test_regression_dump_restore($oldnode, %node_params);
- }
+ test_regression_dump_restore($oldnode, %node_params);
}
# Initialize a new node for the upgrade.
@@ -590,53 +583,34 @@ sub test_regression_dump_restore
$dst_node->append_conf('postgresql.conf', 'autovacuum = off');
$dst_node->start;
- # Test all formats one by one.
- for my $format ('plain', 'tar', 'directory', 'custom')
- {
- my $dump_file = "$tempdir/regression_dump.$format";
- my $restored_db = 'regression_' . $format;
-
- # Use --create in dump and restore commands so that the restored
- # database has the same configurable variable settings as the original
- # database and the plain dumps taken for comparsion do not differ
- # because of locale changes. Additionally this provides test coverage
- # for --create option.
- $src_node->command_ok(
- [
- 'pg_dump', "-F$format", '--no-sync',
- '-d', $src_node->connstr('regression'),
- '--create', '-f', $dump_file
- ],
- "pg_dump on source instance in $format format");
+ my $dump_file = "$tempdir/regression.dump";
- my @restore_command;
- if ($format eq 'plain')
- {
- # Restore dump in "plain" format using `psql`.
- @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
- }
- else
- {
- @restore_command = [
- 'pg_restore', '--create',
- '-d', 'postgres', $dump_file
- ];
- }
- $dst_node->command_ok(@restore_command,
- "restored dump taken in $format format on destination instance");
+ # Use --create in dump and restore commands so that the restored database
+ # has the same configurable variable settings as the original database so
+ # that the plain dumps taken from both the database taken for comparisong do
+ # not differ because of locale changes. Additionally this provides test
+ # coverage for --create option.
+ #
+ # We use directory format which allows dumping and restoring in parallel to
+ # reduce the test's run time.
+ $src_node->command_ok(
+ [
+ 'pg_dump', '-Fd', '-j2', '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance succeeded");
- my $dst_dump =
- get_dump_for_comparison($dst_node, 'regression',
- 'dest_dump.' . $format, 0);
+ $dst_node->command_ok(
+ [ 'pg_restore', '--create', '-j2', '-d', 'postgres', $dump_file ],
+ "restored dump to destination instance");
- compare_files($src_dump, $dst_dump,
- "dump outputs from original and restored regression database (using $format format) match"
- );
+ my $dst_dump = get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump', 0);
- # Rename the restored database so that it is available for debugging in
- # case the test fails.
- $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
- }
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database match"
+ );
}
# Dump database db from the given node in plain format and adjust it for
--
2.34.1
[text/x-patch] 0001-Test-pg_dump-restore-of-regression-objects-20250331.patch (16.5K, ../../CAExHW5vVFtCejh+UYzNxMGSXOfJ_1xwi5aQHQfemqJgFmkyK5Q@mail.gmail.com/3-0001-Test-pg_dump-restore-of-regression-objects-20250331.patch)
download | inline diff:
From aa1c74951b3b557de8330230185fd5f2ee46ecda Mon Sep 17 00:00:00 2001
From: Ashutosh Bapat <[email protected]>
Date: Thu, 27 Jun 2024 10:03:53 +0530
Subject: [PATCH 1/2] Test pg_dump/restore of regression objects
002_pg_upgrade.pl tests pg_upgrade of the regression database left
behind by regression run. Modify it to test dump and restore of the
regression database as well.
Regression database created by regression run contains almost all the
database objects supported by PostgreSQL in various states. Hence the
new testcase covers dump and restore scenarios not covered by individual
dump/restore cases. Till now 002_pg_upgrade only tested dump/restore
through pg_upgrade which only uses binary mode. Many regression tests
mention that they leave objects behind for dump/restore testing but they
are not tested in a non-binary mode. The new testcase closes that
gap.
Testing dump and restore of regression database makes this test run
longer for a relatively smaller benefit. Hence run it only when
explicitly requested by user by specifying "regress_dump_test" in
PG_TEST_EXTRA.
Note For the reviewers:
The new test has uncovered many bugs so far in one year.
1. Introduced by 14e87ffa5c54. Fixed in fd41ba93e4630921a72ed5127cd0d552a8f3f8fc.
2. Introduced by 0413a556990ba628a3de8a0b58be020fd9a14ed0. Reverted in 74563f6b90216180fc13649725179fc119dddeb5.
3. Fixed by d611f8b1587b8f30caa7c0da99ae5d28e914d54f
3. Being discussed on hackers at https://www.postgresql.org/message-id/CAExHW5s47kmubpbbRJzSM-Zfe0Tj2O3GBagB7YAyE8rQ-V24Uw@mail.gmail.com
Author: Ashutosh Bapat
Reviewed by: Michael Pacquire, Daniel Gustafsson, Tom Lane, Alvaro Herrera
Discussion: https://www.postgresql.org/message-id/CAExHW5uF5V=Cjecx3_Z=7xfh4rg2Wf61PT+hfquzjBqouRzQJQ@mail.gmail.com
---
doc/src/sgml/regress.sgml | 12 ++
src/bin/pg_upgrade/t/002_pg_upgrade.pl | 144 ++++++++++++++++-
src/test/perl/Makefile | 2 +
src/test/perl/PostgreSQL/Test/AdjustDump.pm | 167 ++++++++++++++++++++
src/test/perl/meson.build | 1 +
5 files changed, 324 insertions(+), 2 deletions(-)
create mode 100644 src/test/perl/PostgreSQL/Test/AdjustDump.pm
diff --git a/doc/src/sgml/regress.sgml b/doc/src/sgml/regress.sgml
index 0e5e8e8f309..237b974b3ab 100644
--- a/doc/src/sgml/regress.sgml
+++ b/doc/src/sgml/regress.sgml
@@ -357,6 +357,18 @@ make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance libpq_encryption'
</para>
</listitem>
</varlistentry>
+
+ <varlistentry>
+ <term><literal>regress_dump_test</literal></term>
+ <listitem>
+ <para>
+ When enabled, <filename>src/bin/pg_upgrade/t/002_pg_upgrade.pl</filename>
+ tests dump and restore of regression database left behind by the
+ regression run. Not enabled by default because it is time and resource
+ consuming.
+ </para>
+ </listitem>
+ </varlistentry>
</variablelist>
Tests for features that are not supported by the current build
diff --git a/src/bin/pg_upgrade/t/002_pg_upgrade.pl b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
index 00051b85035..8d22d538529 100644
--- a/src/bin/pg_upgrade/t/002_pg_upgrade.pl
+++ b/src/bin/pg_upgrade/t/002_pg_upgrade.pl
@@ -12,6 +12,7 @@ use File::Path qw(rmtree);
use PostgreSQL::Test::Cluster;
use PostgreSQL::Test::Utils;
use PostgreSQL::Test::AdjustUpgrade;
+use PostgreSQL::Test::AdjustDump;
use Test::More;
# Can be changed to test the other modes.
@@ -35,8 +36,8 @@ sub generate_db
"created database with ASCII characters from $from_char to $to_char");
}
-# Filter the contents of a dump before its use in a content comparison.
-# This returns the path to the filtered dump.
+# Filter the contents of a dump before its use in a content comparison for
+# upgrade testing. This returns the path to the filtered dump.
sub filter_dump
{
my ($is_old, $old_version, $dump_file) = @_;
@@ -262,6 +263,21 @@ else
}
}
is($rc, 0, 'regression tests pass');
+
+ # Test dump/restore of the objects left behind by regression. Ideally it
+ # should be done in a separate TAP test, but doing it here saves us one full
+ # regression run.
+ #
+ # This step takes several extra seconds and some extra disk space, so
+ # requires an opt-in with the PG_TEST_EXTRA environment variable.
+ #
+ # Do this while the old cluster is running before it is shut down by the
+ # upgrade test.
+ if ( $ENV{PG_TEST_EXTRA}
+ && $ENV{PG_TEST_EXTRA} =~ /\bregress_dump_test\b/)
+ {
+ test_regression_dump_restore($oldnode, %node_params);
+ }
}
# Initialize a new node for the upgrade.
@@ -539,4 +555,128 @@ my $dump2_filtered = filter_dump(0, $oldnode->pg_version, $dump2_file);
compare_files($dump1_filtered, $dump2_filtered,
'old and new dumps match after pg_upgrade');
+# Test dump and restore of objects left behind by the regression run.
+#
+# It is expected that regression tests, which create 'regression' database, are
+# run on src_node, which in turn, is left in running state. A fresh node is
+# created using given node_params, which are expected to be the same ones use
+# to create src_node, so as to avoid any differences in the databases.
+#
+# Plain dumps from both the nodes are compared to make sure that all the dumped
+# objects are restored faithfully.
+sub test_regression_dump_restore
+{
+ my ($src_node, %node_params) = @_;
+ my $dst_node = PostgreSQL::Test::Cluster->new('dst_node');
+
+ # Make sure that the source and destination nodes have the same version and
+ # do not use custom install paths. In both the cases, the dump files may
+ # require additional adjustments unknown to code here. Do not run this test
+ # in such a case to avoid utilizing the time and resources unnecessarily.
+ if ($src_node->pg_version != $dst_node->pg_version
+ or defined $src_node->{_install_path})
+ {
+ fail("same version dump and restore test using default installation");
+ return;
+ }
+
+ # Dump the original database for comparison later.
+ my $src_dump =
+ get_dump_for_comparison($src_node, 'regression', 'src_dump', 1);
+
+ # Setup destination database cluster
+ $dst_node->init(%node_params);
+ # Stabilize stats for comparison.
+ $dst_node->append_conf('postgresql.conf', 'autovacuum = off');
+ $dst_node->start;
+
+ # Test all formats one by one.
+ for my $format ('plain', 'tar', 'directory', 'custom')
+ {
+ my $dump_file = "$tempdir/regression_dump.$format";
+ my $restored_db = 'regression_' . $format;
+
+ # Use --create in dump and restore commands so that the restored
+ # database has the same configurable variable settings as the original
+ # database and the plain dumps taken for comparsion do not differ
+ # because of locale changes. Additionally this provides test coverage
+ # for --create option.
+ $src_node->command_ok(
+ [
+ 'pg_dump', "-F$format", '--no-sync',
+ '-d', $src_node->connstr('regression'),
+ '--create', '-f', $dump_file
+ ],
+ "pg_dump on source instance in $format format");
+
+ my @restore_command;
+ if ($format eq 'plain')
+ {
+ # Restore dump in "plain" format using `psql`.
+ @restore_command = [ 'psql', '-d', 'postgres', '-f', $dump_file ];
+ }
+ else
+ {
+ @restore_command = [
+ 'pg_restore', '--create',
+ '-d', 'postgres', $dump_file
+ ];
+ }
+ $dst_node->command_ok(@restore_command,
+ "restored dump taken in $format format on destination instance");
+
+ my $dst_dump =
+ get_dump_for_comparison($dst_node, 'regression',
+ 'dest_dump.' . $format, 0);
+
+ compare_files($src_dump, $dst_dump,
+ "dump outputs from original and restored regression database (using $format format) match"
+ );
+
+ # Rename the restored database so that it is available for debugging in
+ # case the test fails.
+ $dst_node->safe_psql('postgres', "ALTER DATABASE regression RENAME TO $restored_db");
+ }
+}
+
+# Dump database db from the given node in plain format and adjust it for
+# comparing dumps from the original and the restored database.
+#
+# file_prefix is used to create unique names for all dump files so that they
+# remain available for debugging in case the test fails.
+#
+# adjust_child_columns is passed to adjust_regress_dumpfile() which actually
+# adjusts the dump output.
+#
+# The name of the file containting adjusted dump is returned.
+sub get_dump_for_comparison
+{
+ my ($node, $db, $file_prefix, $adjust_child_columns) = @_;
+
+ my $dumpfile = $tempdir . '/' . $file_prefix . '.sql';
+ my $dump_adjusted = "${dumpfile}_adjusted";
+
+ # Usually we avoid comparing statistics in our tests since it is flaky by
+ # nature. However, if statistics is dumped and restored it is expected to be
+ # restored as it is i.e. the statistics from the original database and that
+ # from the restored database should match. We turn off autovacuum on the
+ # source and the target database to avoid any statistics update during
+ # restore operation. Hence we do not exclude statistics from dump.
+ $node->command_ok(
+ [
+ 'pg_dump', '--no-sync', '-d', $node->connstr($db), '-f',
+ $dumpfile
+ ],
+ 'dump for comparison succeeded');
+
+ open(my $dh, '>', $dump_adjusted)
+ || die
+ "could not open $dump_adjusted for writing the adjusted dump: $!";
+ print $dh adjust_regress_dumpfile(slurp_file($dumpfile),
+ $adjust_child_columns);
+ close($dh);
+
+ return $dump_adjusted;
+}
+
done_testing();
diff --git a/src/test/perl/Makefile b/src/test/perl/Makefile
index d82fb67540e..def89650ead 100644
--- a/src/test/perl/Makefile
+++ b/src/test/perl/Makefile
@@ -26,6 +26,7 @@ install: all installdirs
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/Cluster.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/BackgroundPsql.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustUpgrade.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ $(INSTALL_DATA) $(srcdir)/PostgreSQL/Test/AdjustDump.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
$(INSTALL_DATA) $(srcdir)/PostgreSQL/Version.pm '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
uninstall:
@@ -36,6 +37,7 @@ uninstall:
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/Cluster.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/BackgroundPsql.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustUpgrade.pm'
+ rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Test/AdjustDump.pm'
rm -f '$(DESTDIR)$(pgxsdir)/$(subdir)/PostgreSQL/Version.pm'
endif
diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
new file mode 100644
index 00000000000..74b9a60cf34
--- /dev/null
+++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
@@ -0,0 +1,167 @@
+
+# Copyright (c) 2024-2025, PostgreSQL Global Development Group
+
+=pod
+
+=head1 NAME
+
+PostgreSQL::Test::AdjustDump - helper module for dump and restore tests
+
+=head1 SYNOPSIS
+
+ use PostgreSQL::Test::AdjustDump;
+
+ # Adjust contents of dump output file so that dump output from original
+ # regression database and that from the restored regression database match
+ $dump = adjust_regress_dumpfile($dump, $adjust_child_columns);
+
+=head1 DESCRIPTION
+
+C<PostgreSQL::Test::AdjustDump> encapsulates various hacks needed to
+compare the results of dump and restore tests
+
+=cut
+
+package PostgreSQL::Test::AdjustDump;
+
+use strict;
+use warnings FATAL => 'all';
+
+use Exporter 'import';
+use Test::More;
+
+our @EXPORT = qw(
+ adjust_regress_dumpfile
+);
+
+=pod
+
+=head1 ROUTINES
+
+=over
+
+=item $dump = adjust_regress_dumpfile($dump, $adjust_child_columns)
+
+If we take dump of the regression database left behind after running regression
+tests, restore the dump, and take dump of the restored regression database, the
+outputs of both the dumps differ in the following cases. This routine adjusts
+the given dump so that dump outputs from the original and restored database,
+respectively, match.
+
+Case 1: Some regression tests purposefully create child tables in such a way
+that the order of their inherited columns differ from column orders of their
+respective parents. In the restored database, however, the order of their
+inherited columns are same as that of their respective parents. Thus the column
+orders of these child tables in the original database and those in the restored
+database differ, causing difference in the dump outputs. See MergeAttributes()
+and dumpTableSchema() for details. This routine rearranges the column
+declarations in the relevant C<CREATE TABLE... INHERITS> statements in the dump
+file from original database to match those from the restored database. We could,
+instead, adjust the statements in the dump from the restored database to match
+those from original database or adjust both to a canonical order. But we have
+chosen to adjust the statements in the dump from original database for no
+particular reason.
+
+Case 2: When dumping COPY statements the columns are ordered by their attribute
+number by fmtCopyColumnList(). If a column is added to a parent table after a
+child has inherited the parent and the child has its own columns, the attribute
+number of the column changes after restoring the child table. This is because
+when executing the dumped C<CREATE TABLE... INHERITS> statement all the parent
+attributes are created before any child attributes. Thus the order of columns in
+COPY statements dumped from the original and the restored databases,
+respectively, differs. Such tables in regression tests are listed below. It is
+hard to adjust the column order in the COPY statement along with the data. Hence
+we just remove such COPY statements from the dump output.
+
+Additionally the routine adjusts blank and new lines to avoid noise.
+
+Note: Usually we avoid comparing statistics in our tests since it is flaky by
+nature. However, if statistics is dumped and restored it is expected to be
+restored as it is i.e. the statistics from the original database and that from
+the restored database should match. Hence we do not filter statistics from dump,
+if it's dumped.
+
+Arguments:
+
+=over
+
+=item C<dump>: Contents of dump file
+
+=item C<adjust_child_columns>: 1 indicates that the given dump file requires
+adjusting columns in the child tables; usually when the dump is from original
+database. 0 indicates no such adjustment is needed; usually when the dump is
+from restored database.
+
+=back
+
+Returns the adjusted dump text.
+
+=cut
+
+sub adjust_regress_dumpfile
+{
+ my ($dump, $adjust_child_columns) = @_;
+
+ # use Unix newlines
+ $dump =~ s/\r\n/\n/g;
+
+ # Adjust the CREATE TABLE ... INHERITS statements.
+ if ($adjust_child_columns)
+ {
+ my $saved_dump = $dump;
+
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_stored_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_stored_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\sgenerated_virtual_tests\.gtestxx_4\s\()
+ (\n\s+b\sinteger),
+ (\n\s+a\sinteger\sNOT\sNULL)/$1$3,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied generated_virtual_tests.gtestxx_4 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c1\s\()
+ (\n\s+int_four\sbigint),
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint)/$1$4,$2,$3/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c1 adjustments');
+
+ $saved_dump = $dump;
+ $dump =~ s/(^CREATE\sTABLE\spublic\.test_type_diff2_c2\s\()
+ (\n\s+int_eight\sbigint),
+ (\n\s+int_two\ssmallint),
+ (\n\s+int_four\sbigint)/$1$3,$4,$2/mgx;
+ ok($saved_dump ne $dump,
+ 'applied public.test_type_diff2_c2 adjustments');
+ }
+
+ # Remove COPY statements with differing column order
+ for my $table (
+ 'public\.b_star', 'public\.c_star',
+ 'public\.cc2', 'public\.d_star',
+ 'public\.e_star', 'public\.f_star',
+ 'public\.renamecolumnanother', 'public\.renamecolumnchild',
+ 'public\.test_type_diff2_c1', 'public\.test_type_diff2_c2',
+ 'public\.test_type_diff_c')
+ {
+ $dump =~ s/^COPY\s$table\s\(.+?^\\\.$//sm;
+ }
+
+ # Suppress blank lines, as some places in pg_dump emit more or fewer.
+ $dump =~ s/\n\n+/\n/g;
+
+ return $dump;
+}
+
+=pod
+
+=back
+
+=cut
+
+1;
diff --git a/src/test/perl/meson.build b/src/test/perl/meson.build
index 58e30f15f9d..492ca571ff8 100644
--- a/src/test/perl/meson.build
+++ b/src/test/perl/meson.build
@@ -14,4 +14,5 @@ install_data(
'PostgreSQL/Test/Cluster.pm',
'PostgreSQL/Test/BackgroundPsql.pm',
'PostgreSQL/Test/AdjustUpgrade.pm',
+ 'PostgreSQL/Test/AdjustDump.pm',
install_dir: dir_pgxs / 'src/test/perl/PostgreSQL/Test')
base-commit: e2809e3a1015697832ee4d37b75ba1cd0caac0f0
--
2.34.1
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 14:22 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-03-28 18:12 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-31 21:17 ` Daniel Gustafsson <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Daniel Gustafsson @ 2025-03-31 21:17 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tom Lane <[email protected]>; Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
> On 28 Mar 2025, at 19:12, Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-28, Tom Lane wrote:
>
>> I think instead of going this direction, we really need to create a
>> separately-purposed script that simply creates "one of everything"
>> without doing anything else (except maybe loading a little data).
>> I believe it'd be a lot easier to remember to add to that when
>> inventing new SQL than to remember to leave something behind from the
>> core regression tests. This would also be far faster to run than any
>> approach that involves picking a random subset of the core test
>> scripts.
>
> FWIW this sounds closely related to what I tried to do with
> src/test/modules/test_ddl_deparse; it's currently incomplete, but maybe
> we can use that as a starting point.
Given where we are in the cycle, it seems to make sense to stick to using the
schedule we already have rather than invent a new process for generating it,
and work on that for 19?
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-28 06:32 ` Ashutosh Bapat <[email protected]>
2025-03-28 09:28 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 10:35 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
1 sibling, 2 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-28 06:32 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 27, 2025 at 10:45 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-27, Ashutosh Bapat wrote:
>
> > On Thu, Mar 27, 2025 at 6:01 PM vignesh C <[email protected]> wrote:
>
> > > Couple of minor thoughts:
> > > 1) I felt this error message is not conveying the error message correctly:
> > > + if ($src_node->pg_version != $dst_node->pg_version
> > > + or defined $src_node->{_install_path})
> > > + {
> > > + fail("same version dump and restore test using default
> > > installation");
> > > + return;
> > > + }
> > >
> > > how about something like below:
> > > fail("source and destination nodes must have the same PostgreSQL
> > > version and default installation paths");
> >
> > The text in ok(), fail() etc. are test names and not error messages.
> > See [1]. Your suggestion and other versions that I came up with became
> > too verbose to be test names. So I think the text here is compromise
> > between conveying enough information and not being too long. We
> > usually have to pick the testname and lookup the test code to
> > investigate the failure. This text serves that purpose.
>
> Maybe
> fail("roundtrip dump/restore of the regression database")
No, that's losing some information like default installation and the
same version.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 06:32 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-28 09:28 ` Ashutosh Bapat <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-28 09:28 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Vignesh and Alvaro
On Fri, Mar 28, 2025 at 12:02 PM Ashutosh Bapat
<[email protected]> wrote:
> >
> > Maybe
> > fail("roundtrip dump/restore of the regression database")
>
> No, that's losing some information like default installation and the
> same version.
How about "dump and restore across servers with same PostgreSQL
version using default installation". That's still mouthful but is more
readable.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 06:32 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-28 10:35 ` Alvaro Herrera <[email protected]>
2025-03-28 11:20 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-28 10:35 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-28, Ashutosh Bapat wrote:
> No, that's losing some information like default installation and the
> same version.
You don't need to preserve such information. This is just a test name.
People looking for more details can grep for the name and they will find
the comments.
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Pido que me den el Nobel por razones humanitarias" (Nicanor Parra)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 06:32 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 10:35 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-28 11:20 ` Ashutosh Bapat <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-28 11:20 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 28, 2025 at 4:05 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-28, Ashutosh Bapat wrote:
>
> > No, that's losing some information like default installation and the
> > same version.
>
> You don't need to preserve such information. This is just a test name.
> People looking for more details can grep for the name and they will find
> the comments.
Ok. In that case what's wrong with the testname I have in the patch?
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
@ 2025-03-21 11:51 ` Ashutosh Bapat <[email protected]>
2025-03-21 12:34 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-21 11:51 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Mar 20, 2025 at 8:37 PM vignesh C <[email protected]> wrote:
>
> Will it help the execution time if we use --jobs in case of pg_dump
> and pg_restore wherever supported:
> + $src_node->command_ok(
> + [
> + 'pg_dump', "-F$format", '--no-sync',
> + '-d', $src_node->connstr('regression'),
> + '--create', '-f', $dump_file
> + ],
> + "pg_dump on source instance in $format format");
> +
> + my @restore_command;
> + if ($format eq 'plain')
> + {
> + # Restore dump in "plain" format using `psql`.
> + @restore_command = [ 'psql', '-d', 'postgres',
> '-f', $dump_file ];
> + }
> + else
> + {
> + @restore_command = [
> + 'pg_restore', '--create',
> + '-d', 'postgres', $dump_file
> + ];
> + }
Will reply to this separately along with reply to Alvaro's comments.
>
> Should the copyright be only 2025 in this case:
> diff --git a/src/test/perl/PostgreSQL/Test/AdjustDump.pm
> b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
> new file mode 100644
> index 00000000000..74b9a60cf34
> --- /dev/null
> +++ b/src/test/perl/PostgreSQL/Test/AdjustDump.pm
> @@ -0,0 +1,167 @@
> +
> +# Copyright (c) 2024-2025, PostgreSQL Global Development Group
The patch was posted in 2024 to this mailing list. So we better
protect the copyright since then. I remember a hackers discussion
where a senior member of the community mentioned that there's not harm
in mentioning longer copyright periods than being stricter about it. I
couldn't find the discussion though.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-21 11:51 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
@ 2025-03-21 12:34 ` Alvaro Herrera <[email protected]>
2025-03-21 12:41 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-03-21 12:34 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Mar-21, Ashutosh Bapat wrote:
> On Thu, Mar 20, 2025 at 8:37 PM vignesh C <[email protected]> wrote:
> > Should the copyright be only 2025 in this case:
> The patch was posted in 2024 to this mailing list. So we better
> protect the copyright since then. I remember a hackers discussion
> where a senior member of the community mentioned that there's not harm
> in mentioning longer copyright periods than being stricter about it. I
> couldn't find the discussion though.
On the other hand, my impression is that we do update copyright years to
current year, when committing new files of patches that have been around
for long.
And there's always
https://liferay.dev/blogs/-/blogs/how-and-why-to-properly-write-copyright-statements-in-your-code
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"Las cosas son buenas o malas segun las hace nuestra opinión" (Lisias)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-21 11:51 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 12:34 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-03-21 12:41 ` Ashutosh Bapat <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-03-21 12:41 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: vignesh C <[email protected]>; Michael Paquier <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Mar 21, 2025 at 6:04 PM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Mar-21, Ashutosh Bapat wrote:
>
> > On Thu, Mar 20, 2025 at 8:37 PM vignesh C <[email protected]> wrote:
>
> > > Should the copyright be only 2025 in this case:
>
> > The patch was posted in 2024 to this mailing list. So we better
> > protect the copyright since then. I remember a hackers discussion
> > where a senior member of the community mentioned that there's not harm
> > in mentioning longer copyright periods than being stricter about it. I
> > couldn't find the discussion though.
>
> On the other hand, my impression is that we do update copyright years to
> current year, when committing new files of patches that have been around
> for long.
>
> And there's always
> https://liferay.dev/blogs/-/blogs/how-and-why-to-properly-write-copyright-statements-in-your-code
Right. So shouldn't the copyright notice be 2024-2025 and not just
only 2025? - Next year it will be changed to 2024-2026.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
@ 2025-04-03 03:59 vignesh C <[email protected]>
0 siblings, 0 replies; 85+ messages in thread
From: vignesh C @ 2025-04-03 03:59 UTC (permalink / raw)
To: Ashutosh Bapat <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Michael Paquier <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 2 Apr 2025 at 13:49, Ashutosh Bapat
<[email protected]> wrote:
>
> On Tue, Apr 1, 2025 at 10:31 PM Alvaro Herrera <[email protected]> wrote:
> >
> > On 2025-Apr-01, Ashutosh Bapat wrote:
> >
> > > Just today morning, I found something which looks like another bug in
> > > statistics dump/restore [1]. As Daniel has expressed upthread [2], we
> > > should go ahead and commit the test even if the bug is not fixed. But
> > > in case it creates a lot of noise and makes the build farm red, we
> > > could suppress the failure by not dumping statistics for comparison
> > > till the bug is fixed. PFA patchset which reintroduces 0003 which
> > > suppresses the statistics dump - in case we think it's needed. I have
> > > made some minor cosmetic changes to 0001 and 0002 as well.
> >
> > I have made some changes of my own, and included --no-statistics.
> > But I had already started messing with your patch, so I didn't look at
> > the cosmetic changes you did here. If they're still relevant, please
> > send them my way.
>
> Thanks a lot. I hope the test will now reveal the problems before they
> are committed :)
>
> You have edited those places anyway. So it's ok.
>
> I have closed the CF entry
> https://commitfest.postgresql.org/patch/4564/ committed. I will
> create another CF entry to park --no-statistics reversal change. That
> way, we will know when statistics dump/restore has become stable.
>
> >
> > Hopefully it won't break, and if it does, it's likely fault of the
> > changes I made. I've run it through CI and all is well though, so
> > fingers crossed.
> > https://cirrus-ci.com/build/6327169669922816
> >
> >
> > I observe in the CI results that the pg_upgrade test is not necessarily
> > the last one to finish. In one case it even finished in place 12!
> >
> > [16:36:48.447] 12/332 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade OK 112.16s 22 subtests passed
> > https://api.cirrus-ci.com/v1/task/5803071017582592/logs/test_world.log
>
> Yes. Few animals that I sampled, the test is finishing pretty early
> even though it's taking longer than many other tests. But it's not the
> longest. I also looked at red animals, but none of them report this
> test to be failing.
I believe this commitfest entry at [1] can be closed now, as the
buildfarm has been running stably for the past few days.
[1] - https://commitfest.postgresql.org/patch/4956/
Regards,
Vignesh
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
@ 2025-04-03 13:40 Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Andres Freund @ 2025-04-03 13:40 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-04-03 10:20:09 +0200, Alvaro Herrera wrote:
> On 2025-Apr-03, Ashutosh Bapat wrote:
>
> > Looks like the problem is in the test itself as pointed out by Jeff in
> > [1]. PFA patch fixing the test and enabling statistics back.
>
> Thanks, pushed.
Since then the pg_upgrade tests have been failing on skink/valgrind, due to
exceeding the already substantially increased timeout.
https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=skink&dt=2025-04-03%2007%3A06%3A19...
(note that there are other issues in that run)
284/333 postgresql:pg_upgrade / pg_upgrade/002_pg_upgrade TIMEOUT 10000.66s killed by signal 15 SIGTERM
[10:38:19.815](16.712s) ok 20 - check that locales in new cluster match original cluster
...
# Running: pg_dumpall --no-sync --dbname port=15114 host=/tmp/bh_AdT5uvQ dbname='postgres' --file /home/bf/bf-build/skink-master/HEAD/pgsql.build/testrun/pg_upgrade/002_pg_upgrade/data/tmp_test_gp2G/dump2.sql
death by signal at /home/bf/bf-build/skink-master/HEAD/pgsql/src/test/perl/PostgreSQL/Test/Cluster.pm line 181.
...
[10:44:11.720](351.905s) # Tests were run but no plan was declared and done_testing() was not seen.
I've increased the timeout even further, but I can't say that I am happy about
the slowest test getting even slower. Adding test time in the serially slowest
test is way worse than adding the same time in a concurrent test.
I suspect that the test will go a bit faster if log_statement weren't forced
on, printing that many log lines, with context, does make valgrind slower,
IME. But Cluster.pm forces it to on, and I suspect that putting a global
log_statement=false into TEMP_CONFIG would have it's own disadvantages.
/me and checks prices for increasing the size of skink's host.
Greetings,
Andres
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
@ 2025-04-04 16:07 ` Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Andres Freund @ 2025-04-04 16:07 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On 2025-04-04 12:01:16 -0400, Andres Freund wrote:
> FWIW, for me 027 is actually considerably faster. In an cassert -O0 build (my
> normal development env, I find even -Og too problematic for debugging):
>
> pg_upgrade/002_pg_upgrade 96.61s
> recovery/027_stream_regress 66.04s
>
> After
> git revert 8806e4e8deb1e21715e031e17181d904825a410e abe56227b2e213755dd3e194c530f5f06467bd7c 172259afb563d35001410dc6daad78b250924038
>
> pg_upgrade/002_pg_upgrade 75.09s
>
> Slower by 29%, far from the 3s increased time I saw mentioned somewhere.
>
>
> And this really affects the overall test time:
>
> All tests before:
> real 1m38.173s
> user 5m52.500s
> sys 4m23.574s
>
> All tests after:
> real 2m0.397s
> user 5m53.625s
> sys 4m30.518s
>
> The CPU time increase is rather minimal, but the wall clock time increase is
> 22%.
>
> 17:
> real 1m14.822s
> user 4m2.630s
> sys 3m22.384s
>
> We regressed wall clock time *60%* from 17->18. Some test cycle increase is
> reasonable and can largely be compensated with hardware, but this cycle we're
> growing way faster than hardware gets faster. I don't think that's
> sustainable.
FWIW, with cassert and -O2, it's:
17:
real 0m53.981s
user 3m22.837s
sys 3m24.237s
HEAD:
real 1m19.749s
user 4m54.526s
sys 4m15.657s
so this isn't just due to me using -O0. A 48% increase is better than a 60%
increase, but it's still not sustainable.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
@ 2025-08-05 14:33 ` Alvaro Herrera <[email protected]>
2025-08-05 14:41 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Alvaro Herrera @ 2025-08-05 14:33 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Ashutosh Bapat <[email protected]>; Daniel Gustafsson <[email protected]>; Tom Lane <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Hello,
On 2025-Apr-04, Andres Freund wrote:
> FWIW, with cassert and -O2, it's:
>
> 17:
> real 0m53.981s
> user 3m22.837s
> sys 3m24.237s
>
> HEAD:
> real 1m19.749s
> user 4m54.526s
> sys 4m15.657s
>
> so this isn't just due to me using -O0. A 48% increase is better than a 60%
> increase, but it's still not sustainable.
I happened to notice that this item was still open in the commitfest,
and rereading the thread I now have second thoughts about having it
enabled by default, giving your complaints about speed. How about
applying this to 18 and master?
--
Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
"This is a foot just waiting to be shot" (Andrew Dunstan)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-08-05 14:41 ` Daniel Gustafsson <[email protected]>
2025-08-05 14:59 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Daniel Gustafsson @ 2025-08-05 14:41 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Tom Lane <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
> On 5 Aug 2025, at 16:33, Alvaro Herrera <[email protected]> wrote:
> I happened to notice that this item was still open in the commitfest,
> and rereading the thread I now have second thoughts about having it
> enabled by default, giving your complaints about speed. How about
> applying this to 18 and master?
Thanks for reviving this. I am +1 on placing this behind PG_TEST_EXTRA as was
discussed upthread.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-08-05 14:41 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
@ 2025-08-05 14:59 ` Tom Lane <[email protected]>
2025-08-05 18:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 85+ messages in thread
From: Tom Lane @ 2025-08-05 14:59 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
Daniel Gustafsson <[email protected]> writes:
> Thanks for reviving this. I am +1 on placing this behind PG_TEST_EXTRA as was
> discussed upthread.
+1 here too.
regards, tom lane
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-08-05 14:41 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-08-05 14:59 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
@ 2025-08-05 18:11 ` Alvaro Herrera <[email protected]>
2025-08-06 01:22 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-08-06 15:10 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
0 siblings, 2 replies; 85+ messages in thread
From: Alvaro Herrera @ 2025-08-05 18:11 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On 2025-Aug-05, Tom Lane wrote:
> Daniel Gustafsson <[email protected]> writes:
> > Thanks for reviving this. I am +1 on placing this behind PG_TEST_EXTRA as was
> > discussed upthread.
>
> +1 here too.
Cool, thanks, done. Now we need a volunteer to set up a buildfarm
animal with this flag ...
--
Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/
"If it is not right, do not do it.
If it is not true, do not say it." (Marcus Aurelius, Meditations)
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-08-05 14:41 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-08-05 14:59 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-08-05 18:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-08-06 01:22 ` Michael Paquier <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Michael Paquier @ 2025-08-06 01:22 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Ashutosh Bapat <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Aug 05, 2025 at 08:11:41PM +0200, Alvaro Herrera wrote:
> Cool, thanks, done. Now we need a volunteer to set up a buildfarm
> animal with this flag ...
Sure. I have added regress_dump_restore to the configuration of
batta, hachi and gokiburi.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 85+ messages in thread
* Re: Test to dump and restore objects left behind by regression
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-08-05 14:41 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-08-05 14:59 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-08-05 18:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
@ 2025-08-06 15:10 ` Ashutosh Bapat <[email protected]>
1 sibling, 0 replies; 85+ messages in thread
From: Ashutosh Bapat @ 2025-08-06 15:10 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Andres Freund <[email protected]>; Michael Paquier <[email protected]>; vignesh C <[email protected]>; Peter Eisentraut <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Aug 6, 2025 at 8:21 AM Alvaro Herrera <[email protected]> wrote:
>
> On 2025-Aug-05, Tom Lane wrote:
>
> > Daniel Gustafsson <[email protected]> writes:
> > > Thanks for reviving this. I am +1 on placing this behind PG_TEST_EXTRA as was
> > > discussed upthread.
> >
> > +1 here too.
>
> Cool, thanks, done. Now we need a volunteer to set up a buildfarm
> animal with this flag ...
Your patch didn't contain doc changes. But the commit has it. Thanks.
--
Best Wishes,
Ashutosh Bapat
^ permalink raw reply [nested|flat] 85+ messages in thread
end of thread, other threads:[~2025-09-29 08:45 UTC | newest]
Thread overview: 85+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-09-25 20:32 [PATCH v5 05/14] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v2 05/11] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v3 05/11] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v2 05/11] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v6 05/15] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v3 05/11] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v7 05/16] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v4 05/12] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2023-09-25 20:32 [PATCH v4 05/12] Improve sentences in overview of system configuration parameters Karl O. Pinc <[email protected]>
2025-01-16 01:27 Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-01-17 17:15 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-01-22 10:58 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-22 11:10 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-02-04 06:06 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-02-19 08:19 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-06-04 13:47 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-06-09 09:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-07-07 10:41 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 11:11 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-03 14:48 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-04 06:20 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-04 14:55 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 03:12 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 10:43 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-11 15:24 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-11 15:32 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Tom Lane <[email protected]>
2025-09-12 18:49 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. [email protected] <[email protected]>
2025-09-12 21:30 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-22 20:05 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-23 19:02 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-24 03:59 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-24 21:40 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-25 14:16 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-09-25 18:50 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-09-29 08:45 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. John Naylor <[email protected]>
2025-01-25 22:07 ` Re: [PATCH] Hex-coding optimizations using SVE on ARM. Nathan Bossart <[email protected]>
2025-03-11 10:44 Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 12:05 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-12 15:58 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-12 16:09 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 06:52 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 08:42 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-13 12:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-13 12:40 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-19 11:43 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-20 15:06 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-20 16:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 12:45 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-21 13:09 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 14:39 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 16:03 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 18:07 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-24 09:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-24 09:59 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-03-24 12:14 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-25 10:39 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 12:31 ` Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-03-27 16:31 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-27 17:15 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 01:36 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-03-28 06:50 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 12:27 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 14:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 14:22 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-03-28 18:12 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-31 11:37 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-31 11:54 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-31 21:17 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-03-28 06:32 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 09:28 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-28 10:35 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-28 11:20 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 11:51 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-03-21 12:34 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-03-21 12:41 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
2025-04-03 03:59 Re: Test to dump and restore objects left behind by regression vignesh C <[email protected]>
2025-04-03 13:40 Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-04-04 16:07 ` Re: Test to dump and restore objects left behind by regression Andres Freund <[email protected]>
2025-08-05 14:33 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-08-05 14:41 ` Re: Test to dump and restore objects left behind by regression Daniel Gustafsson <[email protected]>
2025-08-05 14:59 ` Re: Test to dump and restore objects left behind by regression Tom Lane <[email protected]>
2025-08-05 18:11 ` Re: Test to dump and restore objects left behind by regression Alvaro Herrera <[email protected]>
2025-08-06 01:22 ` Re: Test to dump and restore objects left behind by regression Michael Paquier <[email protected]>
2025-08-06 15:10 ` Re: Test to dump and restore objects left behind by regression Ashutosh Bapat <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox