($INBOX_DIR/description missing)
help / color / mirror / Atom feed[PATCH 03/23] cirrus/ccache: disable compression and show stats
15+ messages / 3 participants
[nested] [flat]
* [PATCH 13/21] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
ci-os-only: linux, windows
ccache since 4.0 enables zstd compression by default.
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
windows has 4.6.1; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should either be shown or logged (or maybe run with CCACHE_NOSTATS,
to avoid re-uploading cache tarball in a 100% cached build, due only to
updating ./**/stats).
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
---
.cirrus.yml | 34 ++++++++++++++++++++++++++++++----
1 file changed, 30 insertions(+), 4 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index e5654bd5589..fd8f27ff593 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "500M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -110,7 +112,11 @@ task:
CXX="ccache c++" \
CFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ ccache --zero-stats
+ su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ ccache --show-stats
upload_caches: ccache
# The use of script avoids make -Otarget complaints about fcntl() on
@@ -206,7 +212,11 @@ task:
CFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all" \
CXXFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all"
EOF
- build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ ccache --zero-stats
+ su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ ccache --show-stats
upload_caches: ccache
test_world_script: |
@@ -325,7 +335,11 @@ task:
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
- build_script: gmake -s -j${BUILD_JOBS} world-bin
+
+ build_script: |
+ ccache --zero-stats
+ gmake -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
upload_caches: ccache
test_world_script: |
@@ -421,7 +435,9 @@ task:
- perl src/tools/msvc/mkvcbuild.pl
build_script:
- vcvarsall x64
+ - ccache --zero-stats
- msbuild %MSBFLAGS% pgsql.sln
+ - ccache --show-stats
always:
upload_caches: ccache
@@ -536,7 +552,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# gcc, cassert on, dtrace off
always:
@@ -547,7 +565,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert off, dtrace off
always:
@@ -557,7 +577,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert on, dtrace on
always:
@@ -569,7 +591,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# cross-compile to windows
always:
@@ -580,7 +604,9 @@ task:
CC="ccache x86_64-w64-mingw32-gcc" \
CXX="ccache x86_64-w64-mingw32-g++"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
###
# Verify docs can be built
--
2.17.1
--STPqjqpCrtky8aYs
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0014-another-way-to-install-uri-regress-uri_regress.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* [PATCH 13/21] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
ci-os-only: linux, windows
ccache since 4.0 enables zstd compression by default.
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
windows has 4.6.1; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should either be shown or logged (or maybe run with CCACHE_NOSTATS,
to avoid re-uploading cache tarball in a 100% cached build, due only to
updating ./**/stats).
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
---
.cirrus.yml | 34 ++++++++++++++++++++++++++++++----
1 file changed, 30 insertions(+), 4 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 50a34373647..1d16594803f 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "500M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -110,7 +112,11 @@ task:
CXX="ccache c++" \
CFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ ccache --zero-stats
+ su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ ccache --show-stats
upload_caches: ccache
# The use of script avoids make -Otarget complaints about fcntl() on
@@ -206,7 +212,11 @@ task:
CFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all" \
CXXFLAGS="-Og -ggdb -fsanitize=undefined,alignment -fno-sanitize-recover=all"
EOF
- build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ ccache --zero-stats
+ su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ ccache --show-stats
upload_caches: ccache
test_world_script: |
@@ -325,7 +335,11 @@ task:
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
- build_script: gmake -s -j${BUILD_JOBS} world-bin
+
+ build_script: |
+ ccache --zero-stats
+ gmake -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
upload_caches: ccache
test_world_script: |
@@ -422,7 +436,9 @@ task:
build_script:
#- ccache --clear
- vcvarsall x64
+ - ccache --zero-stats
- msbuild %MSBFLAGS% pgsql.sln
+ - ccache --show-stats
always:
upload_caches: ccache
@@ -537,7 +553,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# gcc, cassert on, dtrace off
always:
@@ -548,7 +566,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert off, dtrace off
always:
@@ -558,7 +578,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert on, dtrace on
always:
@@ -570,7 +592,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# cross-compile to windows
always:
@@ -581,7 +605,9 @@ task:
CC="ccache x86_64-w64-mingw32-gcc" \
CXX="ccache x86_64-w64-mingw32-g++"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
###
# Verify docs can be built
--
2.17.1
--mln0rGgUGuXEqmuI
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0014-another-way-to-install-uri-regress-uri_regress.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* [PATCH 03/23] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x. But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
windows has 4.6.1; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should either be shown or logged (or maybe run with CCACHE_NOSTATS,
to avoid re-uploading cache tarball in a 100% cached build, due only to
updating ./**/stats).
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
freebsd, linux
ci-os-only: macos
---
.cirrus.yml | 33 ++++++++++++++++++++++++++++-----
1 file changed, 28 insertions(+), 5 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 591e802bee2..54256570030 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "500M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache-stats.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -110,7 +112,10 @@ task:
CXX="ccache c++" \
CFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ build_script: |
+ su postgres -c "ccache --zero-stats"
+ su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ su postgres -c "ccache --show-stats"
upload_caches: ccache
# The use of script avoids make -Otarget complaints about fcntl() on
@@ -206,7 +211,11 @@ task:
CFLAGS="-Og -ggdb" \
CXXFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ su postgres -c "ccache --zero-stats"
+ su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ su postgres -c "ccache --show-stats"
upload_caches: ccache
test_world_script: |
@@ -324,7 +333,11 @@ task:
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
- build_script: gmake -s -j${BUILD_JOBS} world-bin
+
+ build_script: |
+ ccache --zero-stats
+ gmake -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats -v
upload_caches: ccache
test_world_script: |
@@ -482,7 +495,7 @@ task:
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
- CCACHE_MAXSIZE: "1GB"
+ CCACHE_MAXSIZE: "1500MB"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -530,7 +543,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# gcc, cassert on, dtrace off
always:
@@ -541,7 +556,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert off, dtrace off
always:
@@ -551,7 +568,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert on, dtrace on
always:
@@ -563,7 +582,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# cross-compile to windows
always:
@@ -574,7 +595,9 @@ task:
CC="ccache x86_64-w64-mingw32-gcc" \
CXX="ccache x86_64-w64-mingw32-g++"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
###
# Verify docs can be built
--
2.17.1
--tKkaNMvYmhQvRCRK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-cirrus-ccache-add-explicit-cache-keys.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* [PATCH 7/8] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x. But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
XXX windows has 4.7.2; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should be shown and/or logged.
ccache --show-stats shows the *cumulative* stats (including prior
compilations)
ccache --zero-stats clears out not only the global stats, but the
per-file cache stats (from which the global stats are derived) - which
obviously makes the cache work poorly.
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
The log should be written *outside* the ccache folder - it shouldn't be
preserved across cirrusci task invocations.
freebsd, linux, macos
ci-os-only: linux
---
.cirrus.yml | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 221431e70c4..d1c3119e4ff 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "750M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache-stats.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -236,7 +238,7 @@ task:
- name: Linux - Debian Bullseye - Meson
env:
- CCACHE_MAXSIZE: "400M" # tests two different builds
+ CCACHE_MAXSIZE: "1200M" # tests two different builds
configure_script: |
su postgres <<-EOF
@@ -307,6 +309,7 @@ task:
CIRRUS_WORKING_DIR: ${HOME}/pgsql/
CCACHE_DIR: ${HOME}/ccache
+ CCACHE_STATSLOG: ${CCACHE_DIR}.stats.log
HOMEBREW_CACHE: ${HOME}/homebrew-cache
PERL5LIB: ${HOME}/perl5/lib/perl5
@@ -388,7 +391,7 @@ task:
-DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
build
- build_script: ninja -C build -j${BUILD_JOBS}
+ build_script: ninja -C build -j${BUILD_JOBS} && ccache --show-log-stats
upload_caches: ccache
test_world_script: |
@@ -496,14 +499,14 @@ task:
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
- CCACHE_MAXSIZE: "1GB"
+ CCACHE_MAXSIZE: "3G"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
LINUX_MESON_FEATURES: *LINUX_MESON_FEATURES
# task that did not run, count as a success, so we need to recheck Linux'
- # condition here ...
+ # condition here; cirus warns if the "only_if" condition doesn't match the task being depended on
only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*'
container:
--
2.25.1
--rPFbv4B2w2tcwGo5
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0008-cirrus-warnings-use-a-single-common-always-block.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* [PATCH 03/25] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
ccache since 4.0 enables zstd compression by default.
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
windows has 4.6.1; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should either be shown or logged (or maybe run with CCACHE_NOSTATS,
to avoid re-uploading cache tarball in a 100% cached build, due only to
updating ./**/stats).
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
freebsd, linux
ci-os-only: macos
---
.cirrus.yml | 33 ++++++++++++++++++++++++++++-----
1 file changed, 28 insertions(+), 5 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f2861176be2..c0616e8560d 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "500M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache-stats.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -110,7 +112,10 @@ task:
CXX="ccache c++" \
CFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ build_script: |
+ su postgres -c "ccache --zero-stats"
+ su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ su postgres -c "ccache --show-stats"
upload_caches: ccache
# The use of script avoids make -Otarget complaints about fcntl() on
@@ -206,7 +211,11 @@ task:
CFLAGS="-Og -ggdb" \
CXXFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ su postgres -c "ccache --zero-stats"
+ su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ su postgres -c "ccache --show-stats"
upload_caches: ccache
test_world_script: |
@@ -324,7 +333,11 @@ task:
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
- build_script: gmake -s -j${BUILD_JOBS} world-bin
+
+ build_script: |
+ ccache --zero-stats
+ gmake -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats -v
upload_caches: ccache
test_world_script: |
@@ -476,7 +489,7 @@ task:
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
- CCACHE_MAXSIZE: "1GB"
+ CCACHE_MAXSIZE: "1500MB"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -524,7 +537,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# gcc, cassert on, dtrace off
always:
@@ -535,7 +550,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert off, dtrace off
always:
@@ -545,7 +562,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert on, dtrace on
always:
@@ -557,7 +576,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# cross-compile to windows
always:
@@ -568,7 +589,9 @@ task:
CC="ccache x86_64-w64-mingw32-gcc" \
CXX="ccache x86_64-w64-mingw32-g++"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
###
# Verify docs can be built
--
2.17.1
--IS0zKkzwUGydFO0o
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-cirrus-ccache-add-explicit-cache-keys.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* [PATCH 03/23] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x. But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
windows has 4.6.1; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should either be shown or logged (or maybe run with CCACHE_NOSTATS,
to avoid re-uploading cache tarball in a 100% cached build, due only to
updating ./**/stats).
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
freebsd, linux
ci-os-only: macos
---
.cirrus.yml | 33 ++++++++++++++++++++++++++++-----
1 file changed, 28 insertions(+), 5 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 591e802bee2..54256570030 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,9 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "500M"
+ CCACHE_NOCOMPRESS: 1
+ #CCACHE_STATSLOG: ccache-stats.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -110,7 +112,10 @@ task:
CXX="ccache c++" \
CFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ build_script: |
+ su postgres -c "ccache --zero-stats"
+ su postgres -c "gmake -s -j${BUILD_JOBS} world-bin"
+ su postgres -c "ccache --show-stats"
upload_caches: ccache
# The use of script avoids make -Otarget complaints about fcntl() on
@@ -206,7 +211,11 @@ task:
CFLAGS="-Og -ggdb" \
CXXFLAGS="-Og -ggdb"
EOF
- build_script: su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+
+ build_script: |
+ su postgres -c "ccache --zero-stats"
+ su postgres -c "make -s -j${BUILD_JOBS} world-bin"
+ su postgres -c "ccache --show-stats"
upload_caches: ccache
test_world_script: |
@@ -324,7 +333,11 @@ task:
\
LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \
PYTHON=python3
- build_script: gmake -s -j${BUILD_JOBS} world-bin
+
+ build_script: |
+ ccache --zero-stats
+ gmake -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats -v
upload_caches: ccache
test_world_script: |
@@ -482,7 +495,7 @@ task:
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
- CCACHE_MAXSIZE: "1GB"
+ CCACHE_MAXSIZE: "1500MB"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -530,7 +543,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# gcc, cassert on, dtrace off
always:
@@ -541,7 +556,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert off, dtrace off
always:
@@ -551,7 +568,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# clang, cassert on, dtrace on
always:
@@ -563,7 +582,9 @@ task:
${LINUX_CONFIGURE_FEATURES} \
CC="ccache clang" CXX="ccache clang++" CLANG="ccache clang"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
# cross-compile to windows
always:
@@ -574,7 +595,9 @@ task:
CC="ccache x86_64-w64-mingw32-gcc" \
CXX="ccache x86_64-w64-mingw32-g++"
make -s -j${BUILD_JOBS} clean
+ ccache --zero-stats
time make -s -j${BUILD_JOBS} world-bin
+ ccache --show-stats
###
# Verify docs can be built
--
2.17.1
--tKkaNMvYmhQvRCRK
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0004-cirrus-ccache-add-explicit-cache-keys.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* [PATCH 05/11] cirrus/ccache: disable compression and show stats
@ 2022-04-03 05:10 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Justin Pryzby @ 2022-04-03 05:10 UTC (permalink / raw)
Since v4.0, ccache enables zstd compression by default, saving roughly
2x-3x. But, cirrus caches are compressed as tar.gz, so we could disable
ccache compression, allowing cirrus to gzip the uncompressed data
(better than ccache's default of zstd-1).
With default compression enabled (https://cirrus-ci.com/task/6692342840164352):
linux/debian/bullseye has 4.2; 99MB cirrus cache; cache_size_kibibyte 109616
macos has 4.5.1: 47MB cirrus cache; cache_size_kibibyte 52500
freebsd has 3.7.12: 42MB cirrus cache; cache_size_kibibyte 134064
XXX windows has 4.7.2; 180MB cirrus cache; cache_size_kibibyte 51179
todo: compiler warnings
With compression disabled (https://cirrus-ci.com/task/4614182514458624):
linux: 91MB cirrus cache; cache_size_kibibyte 316136
macos: 41MB cirrus cache; cache_size_kibibyte 118068
windows: 42MB cirrus cache; cache_size_kibibyte 134064
freebsd is the same
The stats should be shown and/or logged.
ccache --show-stats shows the *cumulative* stats (including prior
compilations)
ccache --zero-stats clears out not only the global stats, but the
per-file cache stats (from which the global stats are derived) - which
obviously makes the cache work poorly.
Note that ccache 4.4 supports CCACHE_STATSLOG, which seems ideal.
The log should be written *outside* the ccache folder - it shouldn't be
preserved across cirrusci task invocations.
freebsd, linux, macos
ci-os-only: linux
---
.cirrus.yml | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index 47c4caf51ac..d9660149a73 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,11 @@ env:
# Useful to be able to analyse what in a script takes long
CIRRUS_LOG_TIMESTAMP: true
- CCACHE_MAXSIZE: "250M"
+ CCACHE_MAXSIZE: "750M"
+ CCACHE_HARDLINK: 1
+ #CCACHE_NOCOMPRESS: 1
+ # The statslog should be stored somewhere *outside* the persistent ccache dir
+ CCACHE_STATSLOG: ${CIRRUS_WORKING_DIR}/ccache-stats.log
# target to test, for all but windows
CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS
@@ -236,7 +240,7 @@ task:
- name: Linux - Debian Bullseye - Meson
env:
- CCACHE_MAXSIZE: "800M" # tests two different builds
+ CCACHE_MAXSIZE: "2400M" # tests two different builds
configure_script: |
su postgres <<-EOF
@@ -289,7 +293,7 @@ task:
ccache_stats_end_script:
ccache -s
- CCACHE_MAXSIZE=400M ccache --cleanup
+ CCACHE_MAXSIZE=1200M ccache --cleanup
on_failure:
<<: *on_failure_meson
@@ -392,7 +396,7 @@ task:
-DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
build
- build_script: ninja -C build -j${BUILD_JOBS}
+ build_script: ninja -C build -j${BUILD_JOBS} && ccache --show-log-stats
upload_caches: ccache
test_world_script: |
@@ -499,7 +503,7 @@ task:
# flag combinations. The cache size is set high now, and manually pruned later,
# to avoid cache cleaning cycles during the early compilations pruning
# compilation results which could've been used during later compilations.
- CCACHE_MAXSIZE: "2G"
+ CCACHE_MAXSIZE: "6G"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
@@ -620,7 +624,7 @@ task:
always:
ccache_stats_end_script:
ccache -s
- CCACHE_MAXSIZE=1G ccache --cleanup
+ CCACHE_MAXSIZE=3G ccache --cleanup
upload_caches: ccache
--
2.25.1
--I4VOKWutKNZEOIPu
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0006-cirrus-warnings-use-a-single-common-always-block.patch"
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
@ 2023-08-10 08:37 Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Michael Paquier @ 2023-08-10 08:37 UTC (permalink / raw)
To: Masahiro Ikeda <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On Thu, Aug 10, 2023 at 01:08:39PM +0900, Masahiro Ikeda wrote:
> In addition, I change the followings:
> * update about custom wait events in sgml. we don't need to use
> shmem_startup_hook.
> * change the hash names for readability.
> (ex. WaitEventExtensionNameHash -> WaitEventExtensionHashById)
> * fix a bug to fail to get already defined events by names
> because HASH_BLOBS was specified. HASH_STRINGS is right since
> the key is C strings.
That's what I get based on what ShmemInitHash() relies on.
I got a few more comments about v3. Overall this looks much better.
This time the ordering of the operations in WaitEventExtensionNew()
looks much better.
+ * The entry must be stored because it's registered in
+ * WaitEventExtensionNew().
Not sure of the value added by this comment, I would remove it.
+ if (!entry)
+ elog(ERROR, "could not find the name for custom wait event ID %u",
+ eventId);
Or a simpler "could not find custom wait event name for ID %u"?
+static HTAB *WaitEventExtensionHashById; /* find names from ids */
+static HTAB *WaitEventExtensionHashByName; /* find ids from names */
These names are OK here.
+/* Local variables */
+static uint32 worker_spi_wait_event = 0;
That's a cached value, used as a placeholder for the custom wait event
ID found from the table.
+ HASH_ELEM | HASH_STRINGS); /* key is Null-terminated C strings */
Looks obvious to me based on the code, I would remove this note.
+/* hash table entres */
s/entres/entries/
+ /*
+ * Allocate and register a new wait event. But, we need to recheck because
+ * other processes could already do so while releasing the lock.
+ */
Could be reworked for the second sentence, like "Recheck if the event
exists, as it could be possible that a concurrent process has inserted
one with the same name while the lock was previously released."
+ /* Recheck */
Duplicate.
/* OK to make connection */
- conn = libpqsrv_connect(connstr, WAIT_EVENT_EXTENSION);
+ if (wait_event_info == 0)
+ wait_event_info = WaitEventExtensionNew("dblink_get_con");
+ conn = libpqsrv_connect(connstr, wait_event_info);
It is going to be difficult to make that simpler ;)
This looks correct, but perhaps we need to think harder about the
custom event names and define a convention when more of this stuff is
added to the core modules.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../ZNSh41m%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
@ 2023-08-13 23:06 ` Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Michael Paquier @ 2023-08-13 23:06 UTC (permalink / raw)
To: Masahiro Ikeda <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On Thu, Aug 10, 2023 at 05:37:55PM +0900, Michael Paquier wrote:
> This looks correct, but perhaps we need to think harder about the
> custom event names and define a convention when more of this stuff is
> added to the core modules.
Okay, I have put my hands on that, fixing a couple of typos, polishing
a couple of comments, clarifying the docs and applying an indentation.
And here is a v4.
Any thoughts or comments? I'd like to apply that soon, so as we are
able to move on with the wait event catalog and assigning custom wait
events to the other in-core modules.
--
Michael
Attachments:
[text/x-diff] v4-0001-Change-custom-wait-events-to-use-shared-hash-tabl.patch (24.9K, ../../[email protected]/2-v4-0001-Change-custom-wait-events-to-use-shared-hash-tabl.patch)
download | inline diff:
From 83e40dace75d0df91e8b7f617bd03d02bd8450d4 Mon Sep 17 00:00:00 2001
From: Michael Paquier <[email protected]>
Date: Mon, 14 Aug 2023 07:48:15 +0900
Subject: [PATCH v4] Change custom wait events to use shared hash tables
Currently, names of the custom wait event must be registered for each
backends, requiring all these to link to the shared memory area of an
extention, even if not loaded with shared_preload_libraries.
This patch relaxes the constraints by storing the wait events and their
names in hash tables in shared memory. First, this has the advantage to
simplify the registration of custom wait events to a single routine
call that returns an event ID, either existing or incremented:
uint32 WaitEventExtensionNew(const char *wait_event_name);
Any other backend looking at the wait event information, for instance
via pg_stat_activity, is then able to automatically know about the new
event name.
The code changes done in worker_spi show how things are simplified:
- worker_spi_init() is gone.
- No more shmem hooks.
- The custom wait event ID is cached in the process that needs to set
it, with one single call to WaitEventExtensionNew() to retrieve it.
Per suggestion from Andres Freund.
Author: Masahiro Ikeda
Discussion: https://postgr.es/m/[email protected]
---
src/include/utils/wait_event.h | 18 +-
src/backend/storage/lmgr/lwlocknames.txt | 1 +
src/backend/utils/activity/wait_event.c | 217 +++++++++++-------
.../utils/activity/wait_event_names.txt | 1 +
.../modules/worker_spi/t/001_worker_spi.pl | 18 +-
.../modules/worker_spi/worker_spi--1.0.sql | 5 -
src/test/modules/worker_spi/worker_spi.c | 109 +--------
doc/src/sgml/monitoring.sgml | 5 +-
doc/src/sgml/xfunc.sgml | 26 +--
src/tools/pgindent/typedefs.list | 2 +
10 files changed, 164 insertions(+), 238 deletions(-)
diff --git a/src/include/utils/wait_event.h b/src/include/utils/wait_event.h
index aad8bc08fa..3cffae23e1 100644
--- a/src/include/utils/wait_event.h
+++ b/src/include/utils/wait_event.h
@@ -44,12 +44,14 @@ extern PGDLLIMPORT uint32 *my_wait_event_info;
* Use this category when the server process is waiting for some condition
* defined by an extension module.
*
- * Extensions can define their own wait events in this category. First,
- * they should call WaitEventExtensionNew() to get one or more wait event
- * IDs that are allocated from a shared counter. These can be used directly
- * with pgstat_report_wait_start() or equivalent. Next, each individual
- * process should call WaitEventExtensionRegisterName() to associate a wait
- * event string to the number allocated previously.
+ * Extensions can define their own wait events in this category. They should
+ * call WaitEventExtensionNew() with a wait event string. If the wait event
+ * associated the string is already allocated, it returns the wait event info.
+ * If not, it will get one wait event ID that is allocated from a shared
+ + counter, associate the string to the number in the shared dynamic hash and
+ * return the wait event info.
+ *
+ * The ID retrieved can be used with pgstat_report_wait_start() or equivalent.
*/
typedef enum
{
@@ -60,9 +62,7 @@ typedef enum
extern void WaitEventExtensionShmemInit(void);
extern Size WaitEventExtensionShmemSize(void);
-extern uint32 WaitEventExtensionNew(void);
-extern void WaitEventExtensionRegisterName(uint32 wait_event_info,
- const char *wait_event_name);
+extern uint32 WaitEventExtensionNew(const char *wait_event_name);
/* ----------
* pgstat_report_wait_start() -
diff --git a/src/backend/storage/lmgr/lwlocknames.txt b/src/backend/storage/lmgr/lwlocknames.txt
index b34b6afecd..7af3e0ba1a 100644
--- a/src/backend/storage/lmgr/lwlocknames.txt
+++ b/src/backend/storage/lmgr/lwlocknames.txt
@@ -53,3 +53,4 @@ XactTruncationLock 44
# 45 was XactTruncationLock until removal of BackendRandomLock
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
+WaitEventExtensionLock 48
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index b3596ece80..14750233d4 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -45,6 +45,41 @@ uint32 *my_wait_event_info = &local_my_wait_event_info;
#define WAIT_EVENT_CLASS_MASK 0xFF000000
#define WAIT_EVENT_ID_MASK 0x0000FFFF
+/*
+ * Hash tables for storing custom wait event ids and their names in
+ * shared memory.
+ *
+ * WaitEventExtensionHashById is used to find the name from a event id.
+ * Any backend can search it to find custom wait events.
+ *
+ * WaitEventExtensionHashByName is used to find the event ID from a name.
+ * It is used to ensure that no duplicated entries are registered.
+ *
+ * The size of the hash table is based on the assumption that
+ * WAIT_EVENT_EXTENSION_BASH_INIT_SIZE is enough for most cases, and it seems
+ * unlikely that the number of entries will reach
+ * WAIT_EVENT_EXTENSION_BASH_MAX_SIZE.
+ */
+static HTAB *WaitEventExtensionHashById; /* find names from IDs */
+static HTAB *WaitEventExtensionHashByName; /* find IDs from names */
+
+#define WAIT_EVENT_EXTENSION_HASH_INIT_SIZE 16
+#define WAIT_EVENT_EXTENSION_HASH_MAX_SIZE 128
+
+/* hash table entries */
+typedef struct WaitEventExtensionEntryById
+{
+ uint16 event_id; /* hash key */
+ char wait_event_name[NAMEDATALEN]; /* custom wait event name */
+} WaitEventExtensionEntryById;
+
+typedef struct WaitEventExtensionEntryByName
+{
+ char wait_event_name[NAMEDATALEN]; /* hash key */
+ uint16 event_id; /* wait event ID */
+} WaitEventExtensionEntryByName;
+
+
/* dynamic allocation counter for custom wait events in extensions */
typedef struct WaitEventExtensionCounterData
{
@@ -59,36 +94,39 @@ static WaitEventExtensionCounterData *WaitEventExtensionCounter;
#define NUM_BUILTIN_WAIT_EVENT_EXTENSION \
(WAIT_EVENT_EXTENSION_FIRST_USER_DEFINED - WAIT_EVENT_EXTENSION)
-/*
- * This is indexed by event ID minus NUM_BUILTIN_WAIT_EVENT_EXTENSION, and
- * stores the names of all dynamically-created event IDs known to the current
- * process. Any unused entries in the array will contain NULL.
- */
-static const char **WaitEventExtensionNames = NULL;
-static int WaitEventExtensionNamesAllocated = 0;
+/* wait event info for extensions */
+#define WAIT_EVENT_EXTENSION_INFO(eventId) (PG_WAIT_EXTENSION | eventId)
static const char *GetWaitEventExtensionIdentifier(uint16 eventId);
/*
- * Return the space for dynamic allocation counter.
+ * Return the space for dynamic shared hash tables and dynamic allocation counter.
*/
Size
WaitEventExtensionShmemSize(void)
{
- return sizeof(WaitEventExtensionCounterData);
+ Size sz;
+
+ sz = MAXALIGN(sizeof(WaitEventExtensionCounterData));
+ sz = add_size(sz, hash_estimate_size(WAIT_EVENT_EXTENSION_HASH_MAX_SIZE,
+ sizeof(WaitEventExtensionEntryById)));
+ sz = add_size(sz, hash_estimate_size(WAIT_EVENT_EXTENSION_HASH_MAX_SIZE,
+ sizeof(WaitEventExtensionEntryByName)));
+ return sz;
}
/*
- * Allocate shmem space for dynamic allocation counter.
+ * Allocate shmem space for dynamic shared hash and dynamic allocation counter.
*/
void
WaitEventExtensionShmemInit(void)
{
bool found;
+ HASHCTL info;
WaitEventExtensionCounter = (WaitEventExtensionCounterData *)
ShmemInitStruct("WaitEventExtensionCounterData",
- WaitEventExtensionShmemSize(), &found);
+ MAXALIGN(sizeof(WaitEventExtensionCounterData)), &found);
if (!found)
{
@@ -96,21 +134,77 @@ WaitEventExtensionShmemInit(void)
WaitEventExtensionCounter->nextId = NUM_BUILTIN_WAIT_EVENT_EXTENSION;
SpinLockInit(&WaitEventExtensionCounter->mutex);
}
+
+ /* initialize or attach the hash tables to store custom wait events */
+ info.keysize = sizeof(uint16);
+ info.entrysize = sizeof(WaitEventExtensionEntryById);
+ WaitEventExtensionHashById = ShmemInitHash("WaitEventExtension hash by id",
+ WAIT_EVENT_EXTENSION_HASH_INIT_SIZE,
+ WAIT_EVENT_EXTENSION_HASH_MAX_SIZE,
+ &info,
+ HASH_ELEM | HASH_BLOBS);
+
+ info.keysize = sizeof(char[NAMEDATALEN]);
+ info.entrysize = sizeof(WaitEventExtensionEntryByName);
+ /* key is a NULL-terminated string */
+ WaitEventExtensionHashByName = ShmemInitHash("WaitEventExtension hash by name",
+ WAIT_EVENT_EXTENSION_HASH_INIT_SIZE,
+ WAIT_EVENT_EXTENSION_HASH_MAX_SIZE,
+ &info,
+ HASH_ELEM | HASH_STRINGS);
}
/*
- * Allocate a new event ID and return the wait event.
+ * Allocate a new event ID and return the wait event info.
+ *
+ * If the wait event name is already defined, this does not allocate a new
+ * entry; it returns the wait event information associated to the name.
*/
uint32
-WaitEventExtensionNew(void)
+WaitEventExtensionNew(const char *wait_event_name)
{
uint16 eventId;
+ bool found;
+ WaitEventExtensionEntryByName *entry_by_name;
+ WaitEventExtensionEntryById *entry_by_id;
- Assert(LWLockHeldByMeInMode(AddinShmemInitLock, LW_EXCLUSIVE));
+ /* Check the limit of the length of the event name */
+ if (strlen(wait_event_name) >= NAMEDATALEN)
+ elog(ERROR,
+ "cannot use custom wait event string longer than %u characters",
+ NAMEDATALEN - 1);
+ /*
+ * Check the wait event info associated to the name is already defined.
+ * Return it if so.
+ */
+ LWLockAcquire(WaitEventExtensionLock, LW_SHARED);
+ entry_by_name = (WaitEventExtensionEntryByName *)
+ hash_search(WaitEventExtensionHashByName, wait_event_name,
+ HASH_FIND, &found);
+ LWLockRelease(WaitEventExtensionLock);
+ if (found)
+ return WAIT_EVENT_EXTENSION_INFO(entry_by_name->event_id);
+
+ /*
+ * Allocate and register a new wait event. Recheck if the event name
+ * exists, as it could be possible that a concurrent process has inserted
+ * one with the same name while the lock was previously released.
+ */
+ LWLockAcquire(WaitEventExtensionLock, LW_EXCLUSIVE);
+ entry_by_name = (WaitEventExtensionEntryByName *)
+ hash_search(WaitEventExtensionHashByName, wait_event_name,
+ HASH_FIND, &found);
+ if (found)
+ {
+ LWLockRelease(WaitEventExtensionLock);
+ return WAIT_EVENT_EXTENSION_INFO(entry_by_name->event_id);
+ }
+
+ /* Allocate a new event Id */
SpinLockAcquire(&WaitEventExtensionCounter->mutex);
- if (WaitEventExtensionCounter->nextId > PG_UINT16_MAX)
+ if (WaitEventExtensionCounter->nextId >= WAIT_EVENT_EXTENSION_HASH_MAX_SIZE)
{
SpinLockRelease(&WaitEventExtensionCounter->mutex);
ereport(ERROR,
@@ -122,64 +216,23 @@ WaitEventExtensionNew(void)
SpinLockRelease(&WaitEventExtensionCounter->mutex);
- return PG_WAIT_EXTENSION | eventId;
-}
+ /* Register the new wait event */
+ entry_by_id = (WaitEventExtensionEntryById *)
+ hash_search(WaitEventExtensionHashById, &eventId,
+ HASH_ENTER, &found);
+ Assert(!found);
+ strlcpy(entry_by_id->wait_event_name, wait_event_name,
+ sizeof(entry_by_id->wait_event_name));
-/*
- * Register a dynamic wait event name for extension in the lookup table
- * of the current process.
- *
- * This routine will save a pointer to the wait event name passed as an argument,
- * so the name should be allocated in a backend-lifetime context
- * (shared memory, TopMemoryContext, static constant, or similar).
- *
- * The "wait_event_name" will be user-visible as a wait event name, so try to
- * use a name that fits the style for those.
- */
-void
-WaitEventExtensionRegisterName(uint32 wait_event_info,
- const char *wait_event_name)
-{
- uint32 classId;
- uint16 eventId;
+ entry_by_name = (WaitEventExtensionEntryByName *)
+ hash_search(WaitEventExtensionHashByName, wait_event_name,
+ HASH_ENTER, &found);
+ Assert(!found);
+ entry_by_name->event_id = eventId;
- classId = wait_event_info & WAIT_EVENT_CLASS_MASK;
- eventId = wait_event_info & WAIT_EVENT_ID_MASK;
+ LWLockRelease(WaitEventExtensionLock);
- /* Check the wait event class. */
- if (classId != PG_WAIT_EXTENSION)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid wait event class %u", classId));
-
- /* This should only be called for user-defined wait event. */
- if (eventId < NUM_BUILTIN_WAIT_EVENT_EXTENSION)
- ereport(ERROR,
- errcode(ERRCODE_INVALID_PARAMETER_VALUE),
- errmsg("invalid wait event ID %u", eventId));
-
- /* Convert to array index. */
- eventId -= NUM_BUILTIN_WAIT_EVENT_EXTENSION;
-
- /* If necessary, create or enlarge array. */
- if (eventId >= WaitEventExtensionNamesAllocated)
- {
- uint32 newalloc;
-
- newalloc = pg_nextpower2_32(Max(8, eventId + 1));
-
- if (WaitEventExtensionNames == NULL)
- WaitEventExtensionNames = (const char **)
- MemoryContextAllocZero(TopMemoryContext,
- newalloc * sizeof(char *));
- else
- WaitEventExtensionNames =
- repalloc0_array(WaitEventExtensionNames, const char *,
- WaitEventExtensionNamesAllocated, newalloc);
- WaitEventExtensionNamesAllocated = newalloc;
- }
-
- WaitEventExtensionNames[eventId] = wait_event_name;
+ return WAIT_EVENT_EXTENSION_INFO(eventId);
}
/*
@@ -188,23 +241,25 @@ WaitEventExtensionRegisterName(uint32 wait_event_info,
static const char *
GetWaitEventExtensionIdentifier(uint16 eventId)
{
+ bool found;
+ WaitEventExtensionEntryById *entry;
+
/* Built-in event? */
if (eventId < NUM_BUILTIN_WAIT_EVENT_EXTENSION)
return "Extension";
- /*
- * It is a user-defined wait event, so look at WaitEventExtensionNames[].
- * However, it is possible that the name has never been registered by
- * calling WaitEventExtensionRegisterName() in the current process, in
- * which case give up and return "extension".
- */
- eventId -= NUM_BUILTIN_WAIT_EVENT_EXTENSION;
+ /* It is a user-defined wait event, so lookup hash table. */
+ LWLockAcquire(WaitEventExtensionLock, LW_SHARED);
+ entry = (WaitEventExtensionEntryById *)
+ hash_search(WaitEventExtensionHashById, &eventId,
+ HASH_FIND, &found);
+ LWLockRelease(WaitEventExtensionLock);
- if (eventId >= WaitEventExtensionNamesAllocated ||
- WaitEventExtensionNames[eventId] == NULL)
- return "extension";
+ if (!entry)
+ elog(ERROR, "could not find custom wait event name for ID %u",
+ eventId);
- return WaitEventExtensionNames[eventId];
+ return entry->wait_event_name;
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index fcd9d2c63c..f9e01e33b1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -317,6 +317,7 @@ WAIT_EVENT_DOCONLY LogicalRepWorker "Waiting to read or update the state of logi
WAIT_EVENT_DOCONLY XactTruncation "Waiting to execute <function>pg_xact_status</function> or update the oldest transaction ID available to it."
WAIT_EVENT_DOCONLY WrapLimitsVacuum "Waiting to update limits on transaction id and multixact consumption."
WAIT_EVENT_DOCONLY NotifyQueueTail "Waiting to update limit on <command>NOTIFY</command> message storage."
+WAIT_EVENT_DOCONLY WaitEventExtension "Waiting to read or update custom wait events information for extensions."
WAIT_EVENT_DOCONLY XactBuffer "Waiting for I/O on a transaction status SLRU buffer."
WAIT_EVENT_DOCONLY CommitTsBuffer "Waiting for I/O on a commit timestamp SLRU buffer."
diff --git a/src/test/modules/worker_spi/t/001_worker_spi.pl b/src/test/modules/worker_spi/t/001_worker_spi.pl
index c3e7f5fbe6..26b8a49bec 100644
--- a/src/test/modules/worker_spi/t/001_worker_spi.pl
+++ b/src/test/modules/worker_spi/t/001_worker_spi.pl
@@ -39,25 +39,11 @@ $node->poll_query_until('postgres',
$result = $node->safe_psql('postgres', 'SELECT * FROM schema4.counted;');
is($result, qq(total|1), 'dynamic bgworker correctly consumed tuple data');
-# Check the wait event used by the dynamic bgworker. For a session without
-# the state in shared memory known, the default of "extension" is the value
-# waited on.
+# Check the wait event used by the dynamic bgworker.
$result = $node->poll_query_until(
'postgres',
qq[SELECT wait_event FROM pg_stat_activity WHERE backend_type ~ 'worker_spi';],
- 'extension');
-is($result, 1, 'dynamic bgworker has reported "extension" as wait event');
-
-# If the shared memory state is loaded (here with worker_spi_init within
-# the same connection as the one querying pg_stat_activity), the wait
-# event is the custom one.
-# The expected result is a special pattern here with a newline coming from the
-# first query where the shared memory state is set.
-$result = $node->poll_query_until(
- 'postgres',
- qq[SELECT worker_spi_init(); SELECT wait_event FROM pg_stat_activity WHERE backend_type ~ 'worker_spi';],
- qq[
-worker_spi_main]);
+ qq[worker_spi_main]);
is($result, 1,
'dynamic bgworker has reported "worker_spi_main" as wait event');
diff --git a/src/test/modules/worker_spi/worker_spi--1.0.sql b/src/test/modules/worker_spi/worker_spi--1.0.sql
index f13f7e0f98..e9d5b07373 100644
--- a/src/test/modules/worker_spi/worker_spi--1.0.sql
+++ b/src/test/modules/worker_spi/worker_spi--1.0.sql
@@ -7,8 +7,3 @@ CREATE FUNCTION worker_spi_launch(pg_catalog.int4)
RETURNS pg_catalog.int4 STRICT
AS 'MODULE_PATHNAME'
LANGUAGE C;
-
-CREATE FUNCTION worker_spi_init()
-RETURNS VOID STRICT
-AS 'MODULE_PATHNAME'
-LANGUAGE C;
diff --git a/src/test/modules/worker_spi/worker_spi.c b/src/test/modules/worker_spi/worker_spi.c
index c4317351ce..98f8d4194b 100644
--- a/src/test/modules/worker_spi/worker_spi.c
+++ b/src/test/modules/worker_spi/worker_spi.c
@@ -44,33 +44,17 @@
PG_MODULE_MAGIC;
-PG_FUNCTION_INFO_V1(worker_spi_init);
PG_FUNCTION_INFO_V1(worker_spi_launch);
PGDLLEXPORT void worker_spi_main(Datum main_arg) pg_attribute_noreturn();
-/* Shared memory state */
-typedef struct worker_spi_state
-{
- /* the wait event defined during initialization phase */
- uint32 wait_event;
-} worker_spi_state;
-
-static worker_spi_state *wsstate = NULL; /* pointer to shared memory */
-
-static shmem_request_hook_type prev_shmem_request_hook = NULL;
-static shmem_request_hook_type prev_shmem_startup_hook = NULL;
-
-static void worker_spi_shmem_request(void);
-static void worker_spi_shmem_startup(void);
-static void worker_spi_shmem_init(void);
-static Size worker_spi_memsize(void);
-
/* GUC variables */
static int worker_spi_naptime = 10;
static int worker_spi_total_workers = 2;
static char *worker_spi_database = NULL;
+/* value cached, fetched from shared memory */
+static uint32 worker_spi_wait_event_main = 0;
typedef struct worktable
{
@@ -78,63 +62,6 @@ typedef struct worktable
const char *name;
} worktable;
-static void
-worker_spi_shmem_request(void)
-{
- if (prev_shmem_request_hook)
- prev_shmem_request_hook();
-
- RequestAddinShmemSpace(worker_spi_memsize());
-}
-
-static void
-worker_spi_shmem_startup(void)
-{
- if (prev_shmem_startup_hook)
- prev_shmem_startup_hook();
-
- worker_spi_shmem_init();
-}
-
-static Size
-worker_spi_memsize(void)
-{
- return MAXALIGN(sizeof(worker_spi_state));
-}
-
-/*
- * Initialize the shared memory state of worker_spi.
- *
- * This routine allocates a new wait event when called the first time.
- * On follow-up calls, the name of the wait event associated with the
- * existing shared memory state is registered.
- */
-static void
-worker_spi_shmem_init(void)
-{
- bool found;
-
- wsstate = NULL;
-
- /* Create or attach to the shared memory state */
- LWLockAcquire(AddinShmemInitLock, LW_EXCLUSIVE);
- wsstate = ShmemInitStruct("worker_spi State",
- sizeof(worker_spi_state),
- &found);
-
- /* Define a new wait event */
- if (!found)
- wsstate->wait_event = WaitEventExtensionNew();
-
- LWLockRelease(AddinShmemInitLock);
-
- /*
- * Register the wait event in the lookup table of the current process.
- */
- WaitEventExtensionRegisterName(wsstate->wait_event, "worker_spi_main");
- return;
-}
-
/*
* Initialize workspace for a worker process: create the schema if it doesn't
* already exist.
@@ -224,9 +151,6 @@ worker_spi_main(Datum main_arg)
/* We're now ready to receive signals */
BackgroundWorkerUnblockSignals();
- /* Create (if necessary) and attach to our shared memory area. */
- worker_spi_shmem_init();
-
/* Connect to our database */
BackgroundWorkerInitializeConnection(worker_spi_database, NULL, 0);
@@ -268,6 +192,10 @@ worker_spi_main(Datum main_arg)
{
int ret;
+ /* First time, allocate or get the custom wait event */
+ if (worker_spi_wait_event_main == 0)
+ worker_spi_wait_event_main = WaitEventExtensionNew("worker_spi_main");
+
/*
* Background workers mustn't call usleep() or any direct equivalent:
* instead, they may wait on their process latch, which sleeps as
@@ -277,7 +205,7 @@ worker_spi_main(Datum main_arg)
(void) WaitLatch(MyLatch,
WL_LATCH_SET | WL_TIMEOUT | WL_EXIT_ON_PM_DEATH,
worker_spi_naptime * 1000L,
- wsstate->wait_event);
+ worker_spi_wait_event_main);
ResetLatch(MyLatch);
CHECK_FOR_INTERRUPTS();
@@ -406,11 +334,6 @@ _PG_init(void)
MarkGUCPrefixReserved("worker_spi");
- prev_shmem_request_hook = shmem_request_hook;
- shmem_request_hook = worker_spi_shmem_request;
- prev_shmem_startup_hook = shmem_startup_hook;
- shmem_startup_hook = worker_spi_shmem_startup;
-
/* set up common data for all our workers */
memset(&worker, 0, sizeof(worker));
worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
@@ -434,21 +357,6 @@ _PG_init(void)
}
}
-/*
- * Wrapper to initialize a session with the shared memory state
- * used by this module. This is a convenience routine to be able to
- * see the custom wait event stored in shared memory without loading
- * through shared_preload_libraries.
- */
-Datum
-worker_spi_init(PG_FUNCTION_ARGS)
-{
- /* Create (if necessary) and attach to our shared memory area. */
- worker_spi_shmem_init();
-
- PG_RETURN_VOID();
-}
-
/*
* Dynamically launch an SPI worker.
*/
@@ -461,9 +369,6 @@ worker_spi_launch(PG_FUNCTION_ARGS)
BgwHandleStatus status;
pid_t pid;
- /* Create (if necessary) and attach to our shared memory area. */
- worker_spi_shmem_init();
-
memset(&worker, 0, sizeof(worker));
worker.bgw_flags = BGWORKER_SHMEM_ACCESS |
BGWORKER_BACKEND_DATABASE_CONNECTION;
diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml
index f4fc5d814f..70511a2388 100644
--- a/doc/src/sgml/monitoring.sgml
+++ b/doc/src/sgml/monitoring.sgml
@@ -1121,9 +1121,8 @@ SELECT pid, wait_event_type, wait_event FROM pg_stat_activity WHERE wait_event i
<literal>LWLock</literal> types
to the list shown in <xref linkend="wait-event-extension-table"/> and
<xref linkend="wait-event-lwlock-table"/>. In some cases, the name
- assigned by an extension will not be available in all server processes;
- so an <literal>Extension</literal> or <literal>LWLock</literal> wait
- event might be reported as just
+ of <literal>LWLock</literal> assigned by an extension will not be
+ available in all server processes; It might be reported as just
<quote><literal>extension</literal></quote> rather than the
extension-assigned name.
</para>
diff --git a/doc/src/sgml/xfunc.sgml b/doc/src/sgml/xfunc.sgml
index d6345a775b..281c178b0e 100644
--- a/doc/src/sgml/xfunc.sgml
+++ b/doc/src/sgml/xfunc.sgml
@@ -3454,33 +3454,15 @@ if (!ptr)
</sect2>
<sect2 id="xfunc-addin-wait-events">
- <title>Shared Memory and Custom Wait Events</title>
+ <title>Custom Wait Events</title>
<para>
Add-ins can define custom wait events under the wait event type
- <literal>Extension</literal>. The add-in's shared library must be
- preloaded by specifying it in <literal>shared_preload_libraries</literal>,
- and register a <literal>shmem_request_hook</literal> and a
- <literal>shmem_startup_hook</literal> in its
- <function>_PG_init</function> function.
- <literal>shmem_request_hook</literal> can request a shared memory size
- to be later used at startup by calling:
+ <literal>Extension</literal> by calling:
<programlisting>
-void RequestAddinShmemSpace(int size)
-</programlisting>
- </para>
- <para>
- <literal>shmem_startup_hook</literal> can allocate in shared memory
- custom wait events by calling while holding the LWLock
- <function>AddinShmemInitLock</function> to avoid any race conditions:
-<programlisting>
-uint32 WaitEventExtensionNew(void)
-</programlisting>
- Next, each process needs to associate the wait event allocated previously
- to a user-facing custom string, which is something done by calling:
-<programlisting>
-void WaitEventExtensionRegisterName(uint32 wait_event_info, const char *wait_event_name)
+uint32 WaitEventExtensionNew(const char *wait_event_name)
</programlisting>
+ The wait event is associated to a user-facing custom string.
An example can be found in <filename>src/test/modules/worker_spi</filename>
in the PostgreSQL source tree.
</para>
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 66823bc2a7..87feb432a0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2992,6 +2992,8 @@ WaitEventBufferPin
WaitEventClient
WaitEventExtension
WaitEventExtensionCounterData
+WaitEventExtensionEntryById
+WaitEventExtensionEntryByName
WaitEventIO
WaitEventIPC
WaitEventSet
--
2.40.1
[application/pgp-signature] signature.asc (833B, ../../[email protected]/3-signature.asc)
download
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
@ 2023-08-14 03:31 ` Masahiro Ikeda <[email protected]>
2023-08-14 06:28 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Masahiro Ikeda @ 2023-08-14 03:31 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On 2023-08-14 08:06, Michael Paquier wrote:
> On Thu, Aug 10, 2023 at 05:37:55PM +0900, Michael Paquier wrote:
>> This looks correct, but perhaps we need to think harder about the
>> custom event names and define a convention when more of this stuff is
>> added to the core modules.
>
> Okay, I have put my hands on that, fixing a couple of typos, polishing
> a couple of comments, clarifying the docs and applying an indentation.
> And here is a v4.
>
> Any thoughts or comments? I'd like to apply that soon, so as we are
> able to move on with the wait event catalog and assigning custom wait
> events to the other in-core modules.
Thanks! I confirmed the changes, and all tests passed.
Regards,
--
Masahiro Ikeda
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
@ 2023-08-14 06:28 ` Michael Paquier <[email protected]>
2023-08-14 08:55 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Michael Paquier @ 2023-08-14 06:28 UTC (permalink / raw)
To: Masahiro Ikeda <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On Mon, Aug 14, 2023 at 12:31:05PM +0900, Masahiro Ikeda wrote:
> Thanks! I confirmed the changes, and all tests passed.
Okay, cool. I got some extra time today and applied that, with a few
more tweaks.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 06:28 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
@ 2023-08-14 08:55 ` Masahiro Ikeda <[email protected]>
2023-08-14 09:26 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Masahiro Ikeda @ 2023-08-14 08:55 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On 2023-08-14 15:28, Michael Paquier wrote:
> On Mon, Aug 14, 2023 at 12:31:05PM +0900, Masahiro Ikeda wrote:
>> Thanks! I confirmed the changes, and all tests passed.
>
> Okay, cool. I got some extra time today and applied that, with a few
> more tweaks.
Thanks for applying master branch!
> This looks correct, but perhaps we need to think harder about the
> custom event names and define a convention when more of this stuff is
> added to the core modules.
I checked the source code how many functions use WAIT_EVENT_EXTENSION.
There are 3 contrib modules and a test module use WAIT_EVENT_EXTENSION
and
there are 8 places where it is called as an argument.
* dblink
- dblink_get_conn(): the wait event is set until the connection
establishment succeeded
- dblink_connect(): same as above
* autoprewarm
- autoprewarm_main(): the wait event is set until shutdown request is
received
- autoprewarm_main(): the wait event is set until the next dump time
* postgres_fdw
- connect_pg_server(): the wait event is set until connection
establishment succeeded
- pgfdw_get_result(): the wait event is set until the results are
received
- pgfdw_get_cleanup_result(): same as above except for abort cleanup
* test_sh_mq
- wait_for_workers_to_become_ready(): the wait event is set until the
workers become ready
I'm thinking a name like "contrib name + description summary" would
be nice. The "contrib name" is namespace-like and the "description
summary"
is the same as the name of the waiting event name in core. For example,
"DblinkConnect" for dblink. In the same as the core one, I thought the
name
should be the camel case.
BTW, is it better to discuss this in a new thread because other
developers
might be interested in user-facing wait event names? I also would like
to add
documentation on the wait events for each modules, as they are not
mentioned now.
Regards,
--
Masahiro Ikeda
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 06:28 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 08:55 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
@ 2023-08-14 09:26 ` Michael Paquier <[email protected]>
2023-08-15 00:14 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Michael Paquier @ 2023-08-14 09:26 UTC (permalink / raw)
To: Masahiro Ikeda <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On Mon, Aug 14, 2023 at 05:55:42PM +0900, Masahiro Ikeda wrote:
> I'm thinking a name like "contrib name + description summary" would
> be nice. The "contrib name" is namespace-like and the "description summary"
> is the same as the name of the waiting event name in core. For example,
> "DblinkConnect" for dblink. In the same as the core one, I thought the name
> should be the camel case.
Or you could use something more in line with the other in-core wait
events formatted as camel-case, like DblinkConnect, etc.
> BTW, is it better to discuss this in a new thread because other developers
> might be interested in user-facing wait event names? I also would like to
> add documentation on the wait events for each modules, as they are not mentioned
> now.
Saying that, having some documentation on the page of each extension
is mandatory once these can be customized, in my opinion. All that
should be discussed on a new, separate thread, to attract the correct
audience.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 06:28 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 08:55 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 09:26 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
@ 2023-08-15 00:14 ` Masahiro Ikeda <[email protected]>
2023-08-15 00:30 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
0 siblings, 1 reply; 15+ messages in thread
From: Masahiro Ikeda @ 2023-08-15 00:14 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On 2023-08-14 18:26, Michael Paquier wrote:
> On Mon, Aug 14, 2023 at 05:55:42PM +0900, Masahiro Ikeda wrote:
>> I'm thinking a name like "contrib name + description summary" would
>> be nice. The "contrib name" is namespace-like and the "description
>> summary"
>> is the same as the name of the waiting event name in core. For
>> example,
>> "DblinkConnect" for dblink. In the same as the core one, I thought the
>> name
>> should be the camel case.
>
> Or you could use something more in line with the other in-core wait
> events formatted as camel-case, like DblinkConnect, etc.
>
>> BTW, is it better to discuss this in a new thread because other
>> developers
>> might be interested in user-facing wait event names? I also would like
>> to
>> add documentation on the wait events for each modules, as they are not
>> mentioned
>> now.
>
> Saying that, having some documentation on the page of each extension
> is mandatory once these can be customized, in my opinion. All that
> should be discussed on a new, separate thread, to attract the correct
> audience.
OK. I'll make a new patch and start a new thread.
Regards,
--
Masahiro Ikeda
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 15+ messages in thread
* Re: Support to define custom wait events for extensions
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 06:28 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 08:55 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 09:26 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-15 00:14 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
@ 2023-08-15 00:30 ` Michael Paquier <[email protected]>
0 siblings, 0 replies; 15+ messages in thread
From: Michael Paquier @ 2023-08-15 00:30 UTC (permalink / raw)
To: Masahiro Ikeda <[email protected]>; +Cc: Andres Freund <[email protected]>; Bharath Rupireddy <[email protected]>; Tristan Partin <[email protected]>; [email protected]
On Tue, Aug 15, 2023 at 09:14:15AM +0900, Masahiro Ikeda wrote:
> OK. I'll make a new patch and start a new thread.
Cool, thanks!
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 15+ messages in thread
end of thread, other threads:[~2023-08-15 00:30 UTC | newest]
Thread overview: 15+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-04-03 05:10 [PATCH 13/21] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2022-04-03 05:10 [PATCH 13/21] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2022-04-03 05:10 [PATCH 03/23] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2022-04-03 05:10 [PATCH 7/8] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2022-04-03 05:10 [PATCH 03/25] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2022-04-03 05:10 [PATCH 03/23] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2022-04-03 05:10 [PATCH 05/11] cirrus/ccache: disable compression and show stats Justin Pryzby <[email protected]>
2023-08-10 08:37 Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-13 23:06 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 03:31 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 06:28 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-14 08:55 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-14 09:26 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
2023-08-15 00:14 ` Re: Support to define custom wait events for extensions Masahiro Ikeda <[email protected]>
2023-08-15 00:30 ` Re: Support to define custom wait events for extensions Michael Paquier <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox