agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feed[PATCH v1 4/5] Allow to resize shared memory without restart
425+ messages / 2 participants
[nested] [flat]
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v47 4/9] repack code cleanups
@ 2026-03-31 14:48 Álvaro Herrera <alvherre@kurilemu.de>
0 siblings, 0 replies; 425+ messages in thread
From: Álvaro Herrera @ 2026-03-31 14:48 UTC (permalink / raw)
---
src/backend/commands/cluster.c | 163 +++++++++++----------------------
1 file changed, 51 insertions(+), 112 deletions(-)
diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c
index fd0bfacaae6..c0220cb3380 100644
--- a/src/backend/commands/cluster.c
+++ b/src/backend/commands/cluster.c
@@ -2459,18 +2459,8 @@ apply_concurrent_changes(BufFile *file, ChangeContext *chgcxt)
TupleTableSlot *spilled_tuple;
TupleTableSlot *old_update_tuple;
TupleTableSlot *ondisk_tuple;
- MemoryContext apply_cxt;
bool have_old_tuple = false;
-
- /*
- * Use a separate memory context for the tuples and any memory needed to
- * process them.
- *
- * XXX would this be better with GenerationContextCreate?
- */
- apply_cxt = AllocSetContextCreate(TopTransactionContext,
- "REPACK - apply",
- ALLOCSET_DEFAULT_SIZES);
+ MemoryContext oldcxt;
spilled_tuple = MakeSingleTupleTableSlot(RelationGetDescr(rel),
&TTSOpsVirtual);
@@ -2479,6 +2469,8 @@ apply_concurrent_changes(BufFile *file, ChangeContext *chgcxt)
old_update_tuple = MakeSingleTupleTableSlot(RelationGetDescr(rel),
&TTSOpsVirtual);
+ oldcxt = MemoryContextSwitchTo(GetPerTupleMemoryContext(chgcxt->cc_estate));
+
while (true)
{
size_t nread;
@@ -2570,14 +2562,15 @@ apply_concurrent_changes(BufFile *file, ChangeContext *chgcxt)
else
elog(ERROR, "unrecognized kind of change: %d", kind);
- MemoryContextReset(apply_cxt);
+ ResetPerTupleExprContext(chgcxt->cc_estate);
}
/* Cleanup. */
ExecDropSingleTupleTableSlot(spilled_tuple);
ExecDropSingleTupleTableSlot(ondisk_tuple);
ExecDropSingleTupleTableSlot(old_update_tuple);
- MemoryContextDelete(apply_cxt);
+
+ MemoryContextSwitchTo(oldcxt);
}
/*
@@ -2588,27 +2581,17 @@ static void
apply_concurrent_insert(Relation rel, TupleTableSlot *slot,
ChangeContext *chgcxt)
{
- List *recheck;
-
/* Put the tuple in the table, but make sure it won't be decoded */
table_tuple_insert(rel, slot, GetCurrentCommandId(true),
HEAP_INSERT_NO_LOGICAL, NULL);
- /*
- * Update indexes with this new tuple. Because we're merely replaying an
- * action that already happened, we have no use for the recheck list of
- * indexes returned, so just free it. XXX or maybe just leave it?
- */
- recheck = ExecInsertIndexTuples(chgcxt->cc_rri,
- chgcxt->cc_estate,
- 0,
- slot,
- NIL, NULL);
- list_free(recheck);
-
+ /* Update indexes with this new tuple. */
+ ExecInsertIndexTuples(chgcxt->cc_rri,
+ chgcxt->cc_estate,
+ 0,
+ slot,
+ NIL, NULL);
pgstat_progress_incr_param(PROGRESS_REPACK_HEAP_TUPLES_INSERTED, 1);
-
- ResetPerTupleExprContext(chgcxt->cc_estate);
}
/*
@@ -2624,10 +2607,9 @@ apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
TM_FailureData tmfd;
TU_UpdateIndexes update_indexes;
TM_Result res;
- List *recheck;
/*
- * Carry out the update, avoiding logical decoding info.
+ * Carry out the update, skipping logical decoding for it.
*/
res = table_tuple_update(rel, &(ondisk_tuple->tts_tid), spilled_tuple,
GetCurrentCommandId(true),
@@ -2646,13 +2628,11 @@ apply_concurrent_update(Relation rel, TupleTableSlot *spilled_tuple,
if (update_indexes == TU_Summarizing)
flags |= EIIT_ONLY_SUMMARIZING;
- recheck = ExecInsertIndexTuples(chgcxt->cc_rri,
- chgcxt->cc_estate,
- flags,
- spilled_tuple,
- NIL, NULL);
- list_free(recheck);
- ResetPerTupleExprContext(chgcxt->cc_estate);
+ ExecInsertIndexTuples(chgcxt->cc_rri,
+ chgcxt->cc_estate,
+ flags,
+ spilled_tuple,
+ NIL, NULL);
}
pgstat_progress_incr_param(PROGRESS_REPACK_HEAP_TUPLES_UPDATED, 1);
@@ -2665,10 +2645,7 @@ apply_concurrent_delete(Relation rel, TupleTableSlot *slot)
TM_FailureData tmfd;
/*
- * Delete tuple from the new heap.
- *
- * Do it like in simple_heap_delete(), except for 'wal_logical' (and
- * except for 'wait').
+ * Delete tuple from the new heap, skipping logical decoding for it.
*/
res = table_tuple_delete(rel, &(slot->tts_tid),
GetCurrentCommandId(true),
@@ -3009,12 +2986,8 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
char relpersistence;
bool is_system_catalog;
Oid ident_idx_new;
- XLogRecPtr wal_insert_ptr,
- end_of_wal;
- char dummy_rec_data = '\0';
- Relation *ind_refs,
- *ind_refs_p;
- int nind;
+ XLogRecPtr end_of_wal;
+ List *indexrels;
ChangeContext chgcxt;
/* Like in cluster_rel(). */
@@ -3041,24 +3014,23 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
*/
ind_oids_new = build_new_indexes(NewHeap, OldHeap, ind_oids_old);
- /* Find "identity index" on the new relation. */
+ /*
+ * The identity index in the new relation appears in the same relative
+ * position as the corresponding index in the old relation. Find it.
+ */
ident_idx_new = InvalidOid;
- forboth(lc, ind_oids_old, lc2, ind_oids_new)
+ foreach_oid(ind_old, ind_oids_old)
{
- Oid ind_old = lfirst_oid(lc);
- Oid ind_new = lfirst_oid(lc2);
-
if (identIdx == ind_old)
{
- ident_idx_new = ind_new;
+ ident_idx_new = list_nth_oid(ind_oids_new,
+ foreach_current_index(ind_old));
break;
}
}
-
- /* Should not happen, given our lock on the old relation. */
if (!OidIsValid(ident_idx_new))
- ereport(ERROR,
- errmsg("identity index missing on the new relation"));
+ elog(ERROR, "could not find index matching \"%s\" at the new relation",
+ get_rel_name(identIdx));
/* Gather information to apply concurrent changes. */
initialize_change_context(&chgcxt, NewHeap, ident_idx_new);
@@ -3074,8 +3046,7 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
* incomplete page; see GetInsertRecPtr), to minimize the amount of data
* we need to flush while holding exclusive lock on the source table.
*/
- wal_insert_ptr = GetInsertRecPtr();
- XLogFlush(wal_insert_ptr);
+ XLogFlush(GetXLogInsertRecPtr());
end_of_wal = GetFlushRecPtr(NULL);
/*
@@ -3094,10 +3065,9 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
/*
* Lock all indexes now, not only the clustering one: all indexes need to
* have their files swapped. While doing that, store their relation
- * references in an array, to handle predicate locks below.
+ * references in a zero-terminated array, to handle predicate locks below.
*/
- ind_refs_p = ind_refs = palloc_array(Relation, list_length(ind_oids_old));
- nind = 0;
+ indexrels = NIL;
foreach_oid(ind_oid, ind_oids_old)
{
Relation index;
@@ -3105,16 +3075,11 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
index = index_open(ind_oid, AccessExclusiveLock);
/*
- * TODO 1) Do we need to check if ALTER INDEX was executed since the
- * new index was created in build_new_indexes()? 2) Specifically for
- * the clustering index, should check_index_is_clusterable() be called
- * here? (Not sure about the latter: ShareUpdateExclusiveLock on the
- * table probably blocks all commands that affect the result of
- * check_index_is_clusterable().)
+ * Some things about the index may have changed before we locked the
+ * index, such as ALTER INDEX RENAME. We don't need to do anything
+ * here to absorb those changes in the new index.
*/
- *ind_refs_p = index;
- ind_refs_p++;
- nind++;
+ indexrels = lappend(indexrels, index);
}
/*
@@ -3128,40 +3093,18 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
* Tuples and pages of the old heap will be gone, but the heap will stay.
*/
TransferPredicateLocksToHeapRelation(OldHeap);
- /* The same for indexes. */
- for (int i = 0; i < nind; i++)
+ foreach_ptr(RelationData, index, indexrels)
{
- Relation index = ind_refs[i];
-
TransferPredicateLocksToHeapRelation(index);
-
- /*
- * References to indexes on the old relation are not needed anymore,
- * however locks stay till the end of the transaction.
- */
index_close(index, NoLock);
}
- pfree(ind_refs);
+ list_free(indexrels);
/*
- * Flush anything we see in WAL, to make sure that all changes committed
- * while we were waiting for the exclusive lock are available for
- * decoding. This should not be necessary if all backends had
- * synchronous_commit set, but we can't rely on this setting.
- *
- * Unfortunately, GetInsertRecPtr() may lag behind the actual insert
- * position, and GetLastImportantRecPtr() points at the start of the last
- * record rather than at the end. Thus the simplest way to determine the
- * insert position is to insert a dummy record and use its LSN.
- *
- * XXX Consider using GetLastImportantRecPtr() and adding the size of the
- * last record (plus the total size of all the page headers the record
- * spans)?
+ * Flush WAL again, to make sure that all changes committed while we were
+ * waiting for the exclusive lock are available for decoding.
*/
- XLogBeginInsert();
- XLogRegisterData(&dummy_rec_data, 1);
- wal_insert_ptr = XLogInsert(RM_XLOG_ID, XLOG_NOOP);
- XLogFlush(wal_insert_ptr);
+ XLogFlush(GetXLogInsertRecPtr());
end_of_wal = GetFlushRecPtr(NULL);
/*
@@ -3186,10 +3129,7 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
{
Oid ind_old = lfirst_oid(lc);
Oid ind_new = lfirst_oid(lc2);
- Oid mapped_tables[4];
-
- /* Zero out possible results from swapped_relation_files */
- memset(mapped_tables, 0, sizeof(mapped_tables));
+ Oid mapped_tables[4] = {0};
swap_relation_files(ind_old, ind_new,
(old_table_oid == RelationRelationId),
@@ -3200,13 +3140,12 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap,
mapped_tables);
#ifdef USE_ASSERT_CHECKING
-
/*
* Concurrent processing is not supported for system relations, so
* there should be no mapped tables.
*/
for (int i = 0; i < 4; i++)
- Assert(mapped_tables[i] == 0);
+ Assert(!OidIsValid(mapped_tables[i]));
#endif
}
@@ -3256,22 +3195,22 @@ build_new_indexes(Relation NewHeap, Relation OldHeap, List *OldIndexes)
pgstat_progress_update_param(PROGRESS_REPACK_PHASE,
PROGRESS_REPACK_PHASE_REBUILD_INDEX);
- foreach_oid(ind_oid, OldIndexes)
+ foreach_oid(oldindex, OldIndexes)
{
- Oid ind_oid_new;
+ Oid newindex;
char *newName;
Relation ind;
- ind = index_open(ind_oid, ShareUpdateExclusiveLock);
+ ind = index_open(oldindex, ShareUpdateExclusiveLock);
- newName = ChooseRelationName(get_rel_name(ind_oid),
+ newName = ChooseRelationName(get_rel_name(oldindex),
NULL,
"repacknew",
get_rel_namespace(ind->rd_index->indrelid),
false);
- ind_oid_new = index_create_copy(NewHeap, false, ind_oid,
- ind->rd_rel->reltablespace, newName);
- result = lappend_oid(result, ind_oid_new);
+ newindex = index_create_copy(NewHeap, false, oldindex,
+ ind->rd_rel->reltablespace, newName);
+ result = lappend_oid(result, newindex);
index_close(ind, NoLock);
}
--
2.47.3
--2fxmamo6mu2qbxgv
Content-Type: text/x-diff; charset=utf-8
Content-Disposition: attachment;
filename="v47-0005-Use-BulkInsertState-when-copying-data-to-the-new.patch"
^ permalink raw reply [nested|flat] 425+ messages in thread
end of thread, other threads:[~2026-03-31 14:48 UTC | newest]
Thread overview: 425+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2026-03-31 14:48 [PATCH v47 4/9] repack code cleanups Álvaro Herrera <alvherre@kurilemu.de>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox