agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feed[PATCH v1 4/5] Allow to resize shared memory without restart
425+ messages / 2 participants
[nested] [flat]
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 1/1] Alert client when prepared statements are deallocated.
@ 2026-06-03 18:26 Nathan Bossart <nathan@postgresql.org>
0 siblings, 0 replies; 425+ messages in thread
From: Nathan Bossart @ 2026-06-03 18:26 UTC (permalink / raw)
While trying to take our own advice and teach libpq's large object
interface to use prepared statements instead of fast-path function
calls, I discovered an interesting problem: a user could deallocate
prepared statements (e.g., via DISCARD ALL) without libpq knowing,
causing the large object functions to subsequently fail. Of the
various options to deal with this problem, the cleanest one seems
to be to solve the general problem and notify clients when a
prepared statement is deallocated. This allows different layers of
the client stack to notice when their statements are deallocated
and to preemptively re-prepare them before reuse.
This commit adds this behavior via a new protocol extension that
libpq enables by default, but that can be disabled via a connection
parameter. libpq offers the ability to register callback functions
that are executed immediately upon receiving a deallocation
notification, as well as a function that returns whether the server
is configured to send these notifications.
This is primarily intended for use in a follow-up commit that
teaches the libpq large object interface to use prepared
statements, but the new message type and libpq interface may be
useful elsewhere.
Reviewed-by: Jacob Champion <jacob.champion@enterprisedb.com>
Reviewed-by: Zsolt Parragi <zsolt.parragi@percona.com>
Discussion: https://postgr.es/m/ahm_4eOKkkKJ3Gds%40nathan
---
doc/src/sgml/libpq.sgml | 91 +++++++++++++++++++
doc/src/sgml/protocol.sgml | 62 ++++++++++++-
src/backend/commands/prepare.c | 33 +++++++
src/backend/tcop/backend_startup.c | 13 ++-
src/bin/pg_upgrade/dump.c | 6 +-
src/bin/pg_upgrade/server.c | 3 +
src/bin/pg_upgrade/task.c | 3 +
src/bin/pg_upgrade/version.c | 3 +-
src/include/libpq/libpq-be.h | 1 +
src/include/libpq/protocol.h | 1 +
src/interfaces/libpq/exports.txt | 2 +
src/interfaces/libpq/fe-connect.c | 63 +++++++++++++
src/interfaces/libpq/fe-protocol3.c | 56 ++++++++++++
src/interfaces/libpq/fe-trace.c | 10 ++
src/interfaces/libpq/libpq-fe.h | 9 ++
src/interfaces/libpq/libpq-int.h | 6 ++
.../libpq_pipeline/traces/prepared.trace | 1 +
17 files changed, 354 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 7d3c3bb66d8..f76b31264f5 100644
--- a/doc/src/sgml/libpq.sgml
+++ b/doc/src/sgml/libpq.sgml
@@ -2297,6 +2297,20 @@ postgresql://%2Fvar%2Flib%2Fpostgresql/dbname
</listitem>
</varlistentry>
+ <varlistentry id="libpq-connect-report-prep-stmt-dealloc" xreflabel="report_prep_stmt_dealloc">
+ <term><literal>report_prep_stmt_dealloc</literal></term>
+ <listitem>
+ <para>
+ If set to <literal>1</literal> (the default),
+ <application>libpq</application> requests the
+ <link linkend="protocol-extensions"><literal>_pq_.report_prep_stmt_dealloc</literal></link>
+ protocol extension, so that the server reports when prepared statements
+ are deallocated. Set it to <literal>0</literal> to suppress the
+ request.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="libpq-connect-scram-client-key" xreflabel="scram_client_key">
<term><literal>scram_client_key</literal></term>
<listitem>
@@ -7253,6 +7267,83 @@ typedef struct pgNotify
</sect1>
+ <sect1 id="libpq-prepstmt-dealloc">
+ <title>Prepared Statement Deallocation Notifications</title>
+
+ <indexterm zone="libpq-prepstmt-dealloc">
+ <primary>prepared statement deallocation</primary>
+ <secondary>in libpq</secondary>
+ </indexterm>
+
+ <para>
+ The server can notify the client whenever a named prepared statement is
+ deallocated by
+ <link linkend="sql-deallocate"><command>DEALLOCATE</command></link>,
+ <link linkend="sql-discard"><command>DISCARD</command></link>,
+ <xref linkend="libpq-PQclosePrepared"/>, or
+ <xref linkend="libpq-PQsendClosePrepared"/>. This is useful when multiple
+ layers of client code share a connection and one drops a statement another
+ prepared. This behavior is negotiated through the
+ <link linkend="protocol-extensions"><literal>_pq_.report_prep_stmt_dealloc</literal></link>
+ protocol extension, which is only available on servers running
+ <productname>PostgreSQL</productname> 20 and later.
+ By default, <application>libpq</application> requests this protocol
+ extension; this can be disabled with the
+ <xref linkend="libpq-connect-report-prep-stmt-dealloc"/> connection
+ parameter.
+ </para>
+
+ <para>
+ Notifications are delivered to callbacks registered with
+ <function>PQaddPrepStmtDeallocCallback</function>.
+
+ <variablelist>
+ <varlistentry id="libpq-PQaddPrepStmtDeallocCallback">
+ <term><function>PQaddPrepStmtDeallocCallback</function><indexterm><primary>PQaddPrepStmtDeallocCallback</primary></indexterm></term>
+ <listitem>
+ <para>
+ Registers a callback to be invoked immediately upon receiving a prepared
+ statement deallocation notification. The value of
+ <parameter>arg</parameter> is passed unaltered to the callback. The
+ <parameter>name</parameter> argument will contain the name of the
+ deallocated prepared statement, or an empty string if all were
+ deallocated. Callbacks run while <application>libpq</application>
+ processes incoming data, so they must not call any
+ <application>libpq</application> functions on the same
+ <parameter>conn</parameter>, and they must not assume that
+ <parameter>name</parameter> survives after returning (copy it if it is
+ needed later). Returns <literal>1</literal> on success or
+ <literal>0</literal> if the callback could not be registered (e.g., due
+ to running out of memory).
+
+<synopsis>
+typedef void (*PQprepStmtDeallocCallback) (PGconn *conn, void *arg, const char *name);
+
+int PQaddPrepStmtDeallocCallback(PGconn *conn, PQprepStmtDeallocCallback cb, void *arg);
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="libpq-PQprepStmtDeallocReporting">
+ <term><function>PQprepStmtDeallocReporting</function><indexterm><primary>PQprepStmtDeallocReporting</primary></indexterm></term>
+ <listitem>
+ <para>
+ Returns <literal>1</literal> if the server accepted the
+ <literal>_pq_.report_prep_stmt_dealloc</literal> protocol extension.
+ Otherwise, returns <literal>0</literal> to indicate that no prepared
+ statement deallocation notifications will be sent.
+
+<synopsis>
+int PQprepStmtDeallocReporting(PGconn *conn);
+</synopsis>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </sect1>
+
<sect1 id="libpq-copy">
<title>Functions Associated with the <command>COPY</command> Command</title>
diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 49f81676712..70ecc451efd 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -346,8 +346,15 @@
<tbody>
<row>
- <entry namest="last" align="center" valign="middle">
- <emphasis>(No supported protocol extensions are currently defined.)</emphasis>
+ <entry><literal>_pq_.report_prep_stmt_dealloc</literal></entry>
+ <entry><emphasis>none</emphasis></entry>
+ <entry>PostgreSQL 20 and later</entry>
+ <entry>When negotiated, the server sends a
+ <link linkend="protocol-message-formats-PrepStmtDealloc">PrepStmtDealloc</link>
+ message whenever a named prepared statement is deallocated by
+ <link linkend="sql-deallocate"><command>DEALLOCATE</command></link>,
+ <link linkend="sql-discard"><command>DISCARD</command></link>, or a
+ <link linkend="protocol-message-formats-Close">Close</link> message.
</entry>
</row>
</tbody>
@@ -1587,6 +1594,20 @@ SELCT 1/0;<!-- this typo is intentional -->
point in the protocol.
</para>
</note>
+
+ <para>
+ If the client requested the
+ <link linkend="protocol-extensions"><literal>_pq_.report_prep_stmt_dealloc</literal></link>
+ protocol extension, the backend sends a PrepStmtDealloc message whenever a
+ named prepared statement is deallocated by
+ <link linkend="sql-deallocate"><command>DEALLOCATE</command></link>,
+ <link linkend="sql-discard"><command>DISCARD</command></link>, or a
+ <link linkend="protocol-message-formats-Close">Close</link> message. This
+ alerts the client that a statement it prepared is gone (e.g., if another
+ layer of the client stack dropped it) and that it must be re-prepared
+ before reuse. The message carries the deallocated statement's name, or an
+ empty string to mean that all prepared statements were deallocated.
+ </para>
</sect2>
<sect2 id="protocol-flow-canceling-requests">
@@ -5873,6 +5894,43 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;"
</listitem>
</varlistentry>
+ <varlistentry id="protocol-message-formats-PrepStmtDealloc">
+ <term>PrepStmtDealloc (B)</term>
+ <listitem>
+ <variablelist>
+ <varlistentry>
+ <term>Byte1('i')</term>
+ <listitem>
+ <para>
+ Identifies the message as a prepared statement deallocation
+ notification. This is sent only if the client requested the
+ <literal>_pq_.report_prep_stmt_dealloc</literal> protocol extension.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>Int32</term>
+ <listitem>
+ <para>
+ Length of message contents in bytes, including self.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term>String</term>
+ <listitem>
+ <para>
+ The name of the deallocated prepared statement. An empty string
+ indicates that all prepared statements were deallocated.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="protocol-message-formats-Query">
<term>Query (F)</term>
<listitem>
diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c
index 876aad2100a..4e7af002d6d 100644
--- a/src/backend/commands/prepare.c
+++ b/src/backend/commands/prepare.c
@@ -26,6 +26,9 @@
#include "commands/explain_state.h"
#include "commands/prepare.h"
#include "funcapi.h"
+#include "libpq/libpq.h"
+#include "libpq/pqformat.h"
+#include "miscadmin.h"
#include "nodes/nodeFuncs.h"
#include "parser/parse_coerce.h"
#include "parser/parse_collate.h"
@@ -512,6 +515,26 @@ DeallocateQuery(DeallocateStmt *stmt)
DropAllPreparedStatements();
}
+/*
+ * Tell the client that a prepared statement has been deallocated (an empty
+ * string means all of them). Only sent to clients that requested the
+ * _pq_.report_prep_stmt_dealloc protocol extension.
+ */
+static void
+SendStmtDeallocMsg(const char *name)
+{
+ StringInfoData buf;
+
+ if (whereToSendOutput != DestRemote)
+ return;
+ if (!MyProcPort || !MyProcPort->report_prep_stmt_dealloc)
+ return;
+
+ pq_beginmessage(&buf, PqMsg_PrepStmtDealloc);
+ pq_sendstring(&buf, name);
+ pq_endmessage(&buf);
+}
+
/*
* Internal version of DEALLOCATE
*
@@ -532,6 +555,9 @@ DropPreparedStatement(const char *stmt_name, bool showError)
/* Now we can remove the hash table entry */
hash_search(prepared_queries, entry->stmt_name, HASH_REMOVE, NULL);
+
+ /* Alert the client */
+ SendStmtDeallocMsg(stmt_name);
}
}
@@ -543,6 +569,7 @@ DropAllPreparedStatements(void)
{
HASH_SEQ_STATUS seq;
PreparedStatement *entry;
+ bool found = false;
/* nothing cached */
if (!prepared_queries)
@@ -552,12 +579,18 @@ DropAllPreparedStatements(void)
hash_seq_init(&seq, prepared_queries);
while ((entry = hash_seq_search(&seq)) != NULL)
{
+ found = true;
+
/* Release the plancache entry */
DropCachedPlan(entry->plansource);
/* Now we can remove the hash table entry */
hash_search(prepared_queries, entry->stmt_name, HASH_REMOVE, NULL);
}
+
+ /* Alert the client */
+ if (found)
+ SendStmtDeallocMsg("");
}
/*
diff --git a/src/backend/tcop/backend_startup.c b/src/backend/tcop/backend_startup.c
index 25205cee0fa..a9158e86ea1 100644
--- a/src/backend/tcop/backend_startup.c
+++ b/src/backend/tcop/backend_startup.c
@@ -806,12 +806,15 @@ retry:
else if (strncmp(nameptr, "_pq_.", 5) == 0)
{
/*
- * Any option beginning with _pq_. is reserved for use as a
- * protocol-level option, but at present no such options are
- * defined.
+ * Options beginning with _pq_. are protocol extensions.
+ * Recognized ones are handled here; report the rest as
+ * unsupported.
*/
- unrecognized_protocol_options =
- lappend(unrecognized_protocol_options, pstrdup(nameptr));
+ if (strcmp(nameptr, "_pq_.report_prep_stmt_dealloc") == 0)
+ port->report_prep_stmt_dealloc = true;
+ else
+ unrecognized_protocol_options =
+ lappend(unrecognized_protocol_options, pstrdup(nameptr));
}
else
{
diff --git a/src/bin/pg_upgrade/dump.c b/src/bin/pg_upgrade/dump.c
index f47c8d06211..8d542903cb6 100644
--- a/src/bin/pg_upgrade/dump.c
+++ b/src/bin/pg_upgrade/dump.c
@@ -24,7 +24,8 @@ generate_old_dump(void)
"\"%s/pg_dumpall\" %s%s --globals-only --quote-all-identifiers "
"--binary-upgrade %s --no-sync -f \"%s/%s\"",
new_cluster.bindir, cluster_conn_opts(&old_cluster),
- protocol_negotiation_supported(&old_cluster) ? "" : " -d \"max_protocol_version=3.0\"",
+ protocol_negotiation_supported(&old_cluster) ?
+ "" : " -d \"max_protocol_version=3.0 report_prep_stmt_dealloc=0\"",
log_opts.verbose ? "--verbose" : "",
log_opts.dumpdir,
GLOBALS_DUMP_FILE);
@@ -45,7 +46,10 @@ generate_old_dump(void)
appendPQExpBufferStr(&connstr, "dbname=");
appendConnStrVal(&connstr, old_db->db_name);
if (!protocol_negotiation_supported(&old_cluster))
+ {
appendPQExpBufferStr(&connstr, " max_protocol_version=3.0");
+ appendPQExpBufferStr(&connstr, " report_prep_stmt_dealloc=0");
+ }
initPQExpBuffer(&escaped_connstr);
appendShellString(&escaped_connstr, connstr.data);
diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c
index 5d81e4e95b8..841802b2b14 100644
--- a/src/bin/pg_upgrade/server.c
+++ b/src/bin/pg_upgrade/server.c
@@ -72,7 +72,10 @@ get_db_conn(ClusterInfo *cluster, const char *db_name)
appendConnStrVal(&conn_opts, cluster->sockdir);
}
if (!protocol_negotiation_supported(cluster))
+ {
appendPQExpBufferStr(&conn_opts, " max_protocol_version=3.0");
+ appendPQExpBufferStr(&conn_opts, " report_prep_stmt_dealloc=0");
+ }
conn = PQconnectdb(conn_opts.data);
termPQExpBuffer(&conn_opts);
diff --git a/src/bin/pg_upgrade/task.c b/src/bin/pg_upgrade/task.c
index b6eb29e1f3a..4c64bbabdcf 100644
--- a/src/bin/pg_upgrade/task.c
+++ b/src/bin/pg_upgrade/task.c
@@ -189,7 +189,10 @@ start_conn(const ClusterInfo *cluster, UpgradeTaskSlot *slot)
appendConnStrVal(&conn_opts, cluster->sockdir);
}
if (!protocol_negotiation_supported(cluster))
+ {
appendPQExpBufferStr(&conn_opts, " max_protocol_version=3.0");
+ appendPQExpBufferStr(&conn_opts, " report_prep_stmt_dealloc=0");
+ }
slot->conn = PQconnectStart(conn_opts.data);
diff --git a/src/bin/pg_upgrade/version.c b/src/bin/pg_upgrade/version.c
index 047670d4acb..9a86b9a8a55 100644
--- a/src/bin/pg_upgrade/version.c
+++ b/src/bin/pg_upgrade/version.c
@@ -30,7 +30,8 @@ jsonb_9_4_check_applicable(ClusterInfo *cluster)
/*
* Older servers can't support newer protocol versions, so their connection
- * strings will need to lock max_protocol_version to 3.0.
+ * strings will need to lock max_protocol_version to 3.0 and disable prepared
+ * statement deallocation reports.
*/
bool
protocol_negotiation_supported(const ClusterInfo *cluster)
diff --git a/src/include/libpq/libpq-be.h b/src/include/libpq/libpq-be.h
index 921b2daa4ff..1fbf08fc420 100644
--- a/src/include/libpq/libpq-be.h
+++ b/src/include/libpq/libpq-be.h
@@ -151,6 +151,7 @@ typedef struct Port
char *user_name;
char *cmdline_options;
List *guc_options;
+ bool report_prep_stmt_dealloc;
/*
* The startup packet application name, only used here for the "connection
diff --git a/src/include/libpq/protocol.h b/src/include/libpq/protocol.h
index eae8f0e7238..aee0675594d 100644
--- a/src/include/libpq/protocol.h
+++ b/src/include/libpq/protocol.h
@@ -53,6 +53,7 @@
#define PqMsg_FunctionCallResponse 'V'
#define PqMsg_CopyBothResponse 'W'
#define PqMsg_ReadyForQuery 'Z'
+#define PqMsg_PrepStmtDealloc 'i'
#define PqMsg_NoData 'n'
#define PqMsg_PortalSuspended 's'
#define PqMsg_ParameterDescription 't'
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 1e3d5bd5867..9413412e65d 100644
--- a/src/interfaces/libpq/exports.txt
+++ b/src/interfaces/libpq/exports.txt
@@ -211,3 +211,5 @@ PQdefaultAuthDataHook 208
PQfullProtocolVersion 209
appendPQExpBufferVA 210
PQgetThreadLock 211
+PQaddPrepStmtDeallocCallback 212
+PQprepStmtDeallocReporting 213
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index 4272d386e64..1f0219d09a3 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -421,6 +421,10 @@ static const internalPQconninfoOption PQconninfoOptions[] = {
"SSL-Key-Log-File", "D", 64,
offsetof(struct pg_conn, sslkeylogfile)},
+ {"report_prep_stmt_dealloc", NULL, "1", NULL,
+ "Report-Prepared-Statement-Deallocations", "", 1,
+ offsetof(struct pg_conn, report_prep_stmt_dealloc)},
+
/* Terminating entry --- MUST BE LAST */
{NULL, NULL, NULL, NULL,
NULL, NULL, 0}
@@ -708,6 +712,7 @@ pqDropServerData(PGconn *conn)
free(conn->write_err_msg);
conn->write_err_msg = NULL;
conn->oauth_want_retry = false;
+ conn->prepStmtDeallocReporting = false;
/*
* Cancel connections need to retain their be_pid and be_cancel_key across
@@ -5178,6 +5183,11 @@ freePGconn(PGconn *conn)
free(conn->rowBuf);
termPQExpBuffer(&conn->errorMessage);
termPQExpBuffer(&conn->workBuffer);
+ free(conn->report_prep_stmt_dealloc);
+ if (conn->prepStmtDeallocCallbacks)
+ free(conn->prepStmtDeallocCallbacks);
+ if (conn->prepStmtDeallocCallbackArgs)
+ free(conn->prepStmtDeallocCallbackArgs);
free(conn);
}
@@ -8426,3 +8436,56 @@ PQgetThreadLock(void)
Assert(pg_g_threadlock);
return pg_g_threadlock;
}
+
+/*
+ * Registers a callback to be invoked whenever the server reports a prepared
+ * statement deallocation. arg is passed through to the callback unaltered.
+ */
+int
+PQaddPrepStmtDeallocCallback(PGconn *conn, PQprepStmtDeallocCallback cb,
+ void *arg)
+{
+ int i;
+ PQprepStmtDeallocCallback *new_cbs;
+ void **new_args;
+
+ if (!conn)
+ return 0;
+
+ i = conn->nPrepStmtDeallocCallbacks;
+
+ new_cbs = realloc(conn->prepStmtDeallocCallbacks,
+ (i + 1) * sizeof(PQprepStmtDeallocCallback));
+ if (!new_cbs)
+ {
+ libpq_append_conn_error(conn, "out of memory");
+ return 0;
+ }
+ conn->prepStmtDeallocCallbacks = new_cbs;
+
+ new_args = realloc(conn->prepStmtDeallocCallbackArgs,
+ (i + 1) * sizeof(void *));
+ if (!new_args)
+ {
+ libpq_append_conn_error(conn, "out of memory");
+ return 0;
+ }
+ conn->prepStmtDeallocCallbackArgs = new_args;
+
+ new_cbs[i] = cb;
+ new_args[i] = arg;
+ conn->nPrepStmtDeallocCallbacks = i + 1;
+
+ return 1;
+}
+
+/*
+ * Returns true if the server accepted the _pq_.report_prep_stmt_dealloc
+ * extension. If false, no notifications will arrive and the caller must
+ * re-prepare on error.
+ */
+int
+PQprepStmtDeallocReporting(PGconn *conn)
+{
+ return conn && conn->prepStmtDeallocReporting;
+}
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 78ffb1025d0..f5cc361fb73 100644
--- a/src/interfaces/libpq/fe-protocol3.c
+++ b/src/interfaces/libpq/fe-protocol3.c
@@ -61,6 +61,30 @@ static size_t build_startup_packet(const PGconn *conn, char *packet,
const PQEnvironmentOption *options);
+/*
+ * Read a PrepStmtDealloc message and invoke the registered callbacks. Broken
+ * out as a subroutine since it can occur in several places.
+ *
+ * Entry: 'i' message type and length already consumed.
+ * Exit: 0 on success, EOF if not enough data.
+ */
+static int
+getPrepStmtDealloc(PGconn *conn)
+{
+ if (pqGets(&conn->workBuffer, conn))
+ return EOF;
+
+ for (int i = 0; i < conn->nPrepStmtDeallocCallbacks; i++)
+ {
+ PQprepStmtDeallocCallback cb = conn->prepStmtDeallocCallbacks[i];
+ void *arg = conn->prepStmtDeallocCallbackArgs[i];
+
+ cb(conn, arg, conn->workBuffer.data);
+ }
+
+ return 0;
+}
+
/*
* parseInput: if appropriate, parse input data from backend
* until input is exhausted or a stopping state is reached.
@@ -184,6 +208,11 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
}
+ else if (id == PqMsg_PrepStmtDealloc)
+ {
+ if (getPrepStmtDealloc(conn))
+ return;
+ }
else
{
/* Any other case is unexpected and we summarily skip it */
@@ -305,6 +334,10 @@ pqParseInput3(PGconn *conn)
if (getParameterStatus(conn))
return;
break;
+ case PqMsg_PrepStmtDealloc:
+ if (getPrepStmtDealloc(conn))
+ return;
+ break;
case PqMsg_BackendKeyData:
/*
@@ -1545,6 +1578,8 @@ pqGetNegotiateProtocolVersion3(PGconn *conn)
{
found_test_protocol_negotiation = true;
}
+ else if (strcmp(conn->workBuffer.data, "_pq_.report_prep_stmt_dealloc") == 0)
+ conn->prepStmtDeallocReporting = false;
else
{
libpq_append_conn_error(conn, "received invalid protocol negotiation message: server reported an unsupported parameter that was not requested (\"%s\")",
@@ -1906,6 +1941,10 @@ getCopyDataMessage(PGconn *conn)
if (getParameterStatus(conn))
return 0;
break;
+ case PqMsg_PrepStmtDealloc:
+ if (getPrepStmtDealloc(conn))
+ return 0;
+ break;
case PqMsg_CopyData:
return msgLength;
case PqMsg_CopyDone:
@@ -2410,6 +2449,10 @@ pqFunctionCall3(PGconn *conn, Oid fnid,
if (getParameterStatus(conn))
continue;
break;
+ case PqMsg_PrepStmtDealloc:
+ if (getPrepStmtDealloc(conn))
+ continue;
+ break;
default:
/* The backend violates the protocol. */
libpq_append_conn_error(conn, "protocol error: id=0x%x", id);
@@ -2451,6 +2494,15 @@ pqBuildStartupPacket3(PGconn *conn, int *packetlen,
char *startpacket;
size_t len;
+ /*
+ * Initialize prepStmtDeallocReporting based on whether the client
+ * requested the protocol extension. pqGetNegotiateProtocolVersion3()
+ * clears this if the server rejects it.
+ */
+ if (conn->report_prep_stmt_dealloc &&
+ strcmp(conn->report_prep_stmt_dealloc, "1") == 0)
+ conn->prepStmtDeallocReporting = true;
+
len = build_startup_packet(conn, NULL, options);
if (len == 0 || len > INT_MAX)
return NULL;
@@ -2525,6 +2577,10 @@ build_startup_packet(const PGconn *conn, char *packet,
if (conn->client_encoding_initial && conn->client_encoding_initial[0])
ADD_STARTUP_OPTION("client_encoding", conn->client_encoding_initial);
+ /* Ask the server to report prepared statement deallocations. */
+ if (conn->prepStmtDeallocReporting)
+ ADD_STARTUP_OPTION("_pq_.report_prep_stmt_dealloc", "");
+
/*
* Add the test_protocol_negotiation option when greasing, to test that
* servers properly report unsupported protocol options in addition to
diff --git a/src/interfaces/libpq/fe-trace.c b/src/interfaces/libpq/fe-trace.c
index c348b08c39b..401e83d95e8 100644
--- a/src/interfaces/libpq/fe-trace.c
+++ b/src/interfaces/libpq/fe-trace.c
@@ -543,6 +543,13 @@ pqTraceOutput_ParameterStatus(FILE *f, const char *message, int *cursor)
pqTraceOutputString(f, message, cursor, false);
}
+static void
+pqTraceOutput_PrepStmtDealloc(FILE *f, const char *message, int *cursor)
+{
+ fprintf(f, "PrepStmtDealloc\t");
+ pqTraceOutputString(f, message, cursor, false);
+}
+
static void
pqTraceOutput_ParameterDescription(FILE *f, const char *message, int *cursor, bool regress)
{
@@ -793,6 +800,9 @@ pqTraceOutputMessage(PGconn *conn, const char *message, bool toServer)
else
pqTraceOutput_ParameterStatus(conn->Pfdebug, message, &logCursor);
break;
+ case PqMsg_PrepStmtDealloc:
+ pqTraceOutput_PrepStmtDealloc(conn->Pfdebug, message, &logCursor);
+ break;
case PqMsg_ParameterDescription:
pqTraceOutput_ParameterDescription(conn->Pfdebug, message, &logCursor, regress);
break;
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index 8ecb9b4a4c7..6cec59c939c 100644
--- a/src/interfaces/libpq/libpq-fe.h
+++ b/src/interfaces/libpq/libpq-fe.h
@@ -486,6 +486,15 @@ typedef void (*pgthreadlock_t) (int acquire);
extern pgthreadlock_t PQregisterThreadLock(pgthreadlock_t newhandler);
extern pgthreadlock_t PQgetThreadLock(void);
+/* callbacks for prepared statement deallocation notifications */
+typedef void (*PQprepStmtDeallocCallback) (PGconn *conn, void *arg,
+ const char *name);
+
+extern int PQaddPrepStmtDeallocCallback(PGconn *conn,
+ PQprepStmtDeallocCallback cb,
+ void *arg);
+extern int PQprepStmtDeallocReporting(PGconn *conn);
+
/* === in fe-trace.c === */
extern void PQtrace(PGconn *conn, FILE *debug_port);
extern void PQuntrace(PGconn *conn);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index 461b39620c3..c4f264cf603 100644
--- a/src/interfaces/libpq/libpq-int.h
+++ b/src/interfaces/libpq/libpq-int.h
@@ -432,6 +432,7 @@ struct pg_conn
char *scram_client_key; /* base64-encoded SCRAM client key */
char *scram_server_key; /* base64-encoded SCRAM server key */
char *sslkeylogfile; /* where should the client write ssl keylogs */
+ char *report_prep_stmt_dealloc; /* request pstmt dealloc reports */
bool cancelRequest; /* true if this connection is used to send a
* cancel request, instead of being a normal
@@ -532,6 +533,11 @@ struct pg_conn
void (*cleanup_async_auth) (PGconn *conn);
pgsocket altsock; /* alternative socket for client to poll */
+ /* prepared statement deallocation notifications */
+ bool prepStmtDeallocReporting;
+ PQprepStmtDeallocCallback *prepStmtDeallocCallbacks;
+ void **prepStmtDeallocCallbackArgs;
+ int nPrepStmtDeallocCallbacks;
/* Transient state needed while establishing connection */
PGTargetServerType target_server_type; /* desired session properties */
diff --git a/src/test/modules/libpq_pipeline/traces/prepared.trace b/src/test/modules/libpq_pipeline/traces/prepared.trace
index aeb5de109e0..ef383f20697 100644
--- a/src/test/modules/libpq_pipeline/traces/prepared.trace
+++ b/src/test/modules/libpq_pipeline/traces/prepared.trace
@@ -7,6 +7,7 @@ B 113 RowDescription 4 "?column?" NNNN 0 NNNN 4 -1 0 "?column?" NNNN 0 NNNN 655
B 5 ReadyForQuery I
F 16 Close S "select_one"
F 4 Sync
+B 15 PrepStmtDealloc "select_one"
B 4 CloseComplete
B 5 ReadyForQuery I
F 16 Describe S "select_one"
--
2.50.1 (Apple Git-155)
--qzBwIkMoeP65cCz/--
^ permalink raw reply [nested|flat] 425+ messages in thread
end of thread, other threads:[~2026-06-03 18:26 UTC | newest]
Thread overview: 425+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2026-06-03 18:26 [PATCH v4 1/1] Alert client when prepared statements are deallocated. Nathan Bossart <nathan@postgresql.org>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox