agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feed[PATCH v1 4/5] Allow to resize shared memory without restart
425+ messages / 2 participants
[nested] [flat]
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v1 4/5] Allow to resize shared memory without restart
@ 2024-10-16 18:24 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2024-10-16 18:24 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Size for every shared memory slot is recalculated based on the new
NBuffers, and extended using mremap. After allocating new space, new
shared structures (buffer blocks, descriptors, etc) are allocated as
needed. Here is how it looks like after raising shared_buffers from 128
MB to 512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 5/6] Allow to resize shared memory without restart
@ 2025-02-20 20:12 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-02-20 20:12 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a global
Barrier to coordinate backends that simultaneously change
shared_buffers, and pieces in shared memory to coordinate backends that
are too late to the party for some reason.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier. Afterwards it does shared memory
resize itself.
* All the backends, that participate in ProcSignal mechanism,
recalculate shared memory size based on the new NBuffers and extend it
using mremap.
* When finished, a backend waits on a global ShmemControl barrier,
untill all backends will be finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all backends are using new value, one backend will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f5a2bd04000-7f5a32e52000 /dev/zero (deleted)
7f5a39252000-7f5a4030e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d7ba000 /dev/zero (deleted)
7f5a53bba000-7f5a5ad26000 /dev/zero (deleted)
7f5a9ad26000-7f5aa9d94000 /dev/zero (deleted)
^ buffers mapping, ~240 MB
7f5d29d94000-7f5d30e00000 /dev/zero (deleted)
-- 512 MB
7f5a2bd04000-7f5a33274000 /dev/zero (deleted)
7f5a39252000-7f5a4057e000 /dev/zero (deleted)
7f5a4670e000-7f5a4d9fa000 /dev/zero (deleted)
7f5a53bba000-7f5a5b1a6000 /dev/zero (deleted)
7f5a9ad26000-7f5ac1f14000 /dev/zero (deleted)
^ buffers mapping, ~625 MB
7f5d29d94000-7f5d30f80000 /dev/zero (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v4 6/8] Allow to resize shared memory without restart
@ 2025-04-06 14:47 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-04-06 14:47 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers and extend it using mremap. One elected process signals the
postmaster to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f90cde00000-7f90d4fa6000 /dev/zero (deleted)
7f90d4fa6000-7f914de00000
7f914de00000-7f915cfa8000 /dev/zero (deleted)
^ buffers mapping, ~241 MB
7f915cfa8000-7f944de00000
7f944de00000-7f94550a8000 /dev/zero (deleted)
7f94550a8000-7f94cde00000
7f94cde00000-7f94d4fe8000 /dev/zero (deleted)
7f94d4fe8000-7f954de00000
7f954de00000-7f9554ff6000 /dev/zero (deleted)
7f9554ff6000-7f958de00000
7f958de00000-7f959508a000 /dev/zero (deleted)
7f959508a000-7f95cde00000
-- 512 MB
7f90cde00000-7f90d5126000 /dev/zero (deleted)
7f90d5126000-7f914de00000
7f914de00000-7f9175128000 /dev/zero (deleted)
^ buffers mapping, ~627 MB
7f9175128000-7f944de00000
7f944de00000-7f9455528000 /dev/zero (deleted)
7f9455528000-7f94cde00000
7f94cde00000-7f94d5228000 /dev/zero (deleted)
7f94d5228000-7f954de00000
7f954de00000-7f9555266000 /dev/zero (deleted)
7f9555266000-7f958de00000
7f958de00000-7f95954aa000 /dev/zero (deleted)
7f95954aa000-7f95cde00000
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v5 07/10] Allow to resize shared memory without restart
@ 2025-06-17 12:16 Dmitrii Dolgov <9erthalion6@gmail.com>
0 siblings, 0 replies; 425+ messages in thread
From: Dmitrii Dolgov @ 2025-06-17 12:16 UTC (permalink / raw)
Add assing hook for shared_buffers to resize shared memory using space,
introduced in the previous commits without requiring PostgreSQL restart.
Essentially the implementation is based on two mechanisms: a
ProcSignalBarrier is used to make sure all processes are starting the
resize procedure simultaneously, and a global Barrier is used to
coordinate after that and make sure all finished processes are waiting
for others that are in progress.
The resize process looks like this:
* The GUC assign hook sets a flag to let the Postmaster know that resize
was requested.
* Postmaster verifies the flag in the event loop, and starts the resize
by emitting a ProcSignal barrier.
* All processes, that participate in ProcSignal mechanism, begin to
process ProcSignal barrier. First a process waits until all processes
have confirmed they received the message and can start simultaneously.
* Every process recalculates shared memory size based on the new
NBuffers, adjusts its size using ftruncate and adjust reservation
permissions with mprotect. One elected process signals the postmaster
to do the same.
* When finished, every process waits on a global ShmemControl barrier,
untill all others are finished as well. This way we ensure three
stages with clear boundaries: before the resize, when all processes
use old NBuffers; during the resize, when processes have mix of old
and new NBuffers, and wait until it's done; after the resize, when all
processes use new NBuffers.
* After all processes are using new value, one of them will initialize
new shared structures (buffer blocks, descriptors, etc) as needed and
broadcast new value of NBuffers via ShmemControl in shared memory.
Other backends are waiting for this operation to finish as well. Then
the barrier is lifted and everything goes as usual.
Since resizing takes time, we need to take into account that during that time:
- New backends can be spawned. They will check status of the barrier
early during the bootstrap, and wait until everything is over to work
with the new NBuffers value.
- Old backends can exit before attempting to resize. Synchronization
used between backends relies on ProcSignalBarrier and waits for all
participants received the message at the beginning to gather all
existing backends.
- Some backends might be blocked and not responsing either before or
after receiving the message. In the first case such backend still
have ProcSignalSlot and should be waited for, in the second case
shared barrier will make sure we still waiting for those backends. In
any case there is an unbounded wait.
- Backends might join barrier in disjoint groups with some time in
between. That means that relying only on the shared dynamic barrier is
not enough -- it will only synchronize resize procedure withing those
groups. That's why we wait first for all participants of ProcSignal
mechanism who received the message.
Here is how it looks like after raising shared_buffers from 128 MB to
512 MB and calling pg_reload_conf():
-- 128 MB
7f87909fc000-7f8798248000 rw-s /memfd:strategy (deleted)
7f8798248000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a4e84000 rw-s /memfd:checkpoint (deleted)
7f87a4e84000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1b42000 rw-s /memfd:iocv (deleted)
7f87b1b42000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cb59c000 rw-s /memfd:descriptors (deleted)
7f87cb59c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f87ece38000 rw-s /memfd:buffers (deleted)
^ buffers content, ~247 MB
7f87ece38000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~2210 MB
7f8877066000-7f887e7d0000 rw-s /memfd:main (deleted)
7f887e7d0000-7f8890a00000 ---s /memfd:main (deleted)
-- 512 MB
7f87909fc000-7f879866a000 rw-s /memfd:strategy (deleted)
7f879866a000-7f879d6ca000 ---s /memfd:strategy (deleted)
7f879d6ca000-7f87a50f4000 rw-s /memfd:checkpoint (deleted)
7f87a50f4000-7f87aa398000 ---s /memfd:checkpoint (deleted)
7f87aa398000-7f87b1d82000 rw-s /memfd:iocv (deleted)
7f87b1d82000-7f87c3d32000 ---s /memfd:iocv (deleted)
7f87c3d32000-7f87cba1c000 rw-s /memfd:descriptors (deleted)
7f87cba1c000-7f87dd6cc000 ---s /memfd:descriptors (deleted)
7f87dd6cc000-7f8804fb8000 rw-s /memfd:buffers (deleted)
^ buffers content, ~632 MB
7f8804fb8000-7f8877066000 ---s /memfd:buffers (deleted)
^ reserved space, ~1824 MB
7f8877066000-7f887e950000 rw-s /memfd:main (deleted)
7f887e950000-7f8890a00000 ---s /memfd:main (deleted)
The implementation supports only increasing of shared_buffers. For
decreasing the value a similar procedure is needed. But the buffer
blocks with data have to be drained first, so that the actual data set
fits into the new smaller space.
^ permalink raw reply [nested|flat] 425+ messages in thread
* [PATCH v2 3/4] Optimize more data type for less memory copy with optional context.
@ 2026-06-02 09:37 Andy Fan <zhihuifan1213@163.com>
0 siblings, 0 replies; 425+ messages in thread
From: Andy Fan @ 2026-06-02 09:37 UTC (permalink / raw)
---
src/backend/access/common/printtup.c | 8 ++-
src/backend/utils/adt/date.c | 32 +++++++----
src/backend/utils/adt/float.c | 62 ++++++++++++++++-----
src/backend/utils/adt/int.c | 82 +++++++++++++++-------------
src/backend/utils/adt/int8.c | 40 ++++++++++----
src/backend/utils/adt/numeric.c | 46 ++++++++++++----
src/backend/utils/adt/timestamp.c | 50 ++++++++++++-----
src/backend/utils/adt/varlena.c | 52 ++++++++----------
src/include/libpq/pqformat.h | 61 +++++++++++++++++++++
src/include/utils/builtins.h | 24 ++++++++
10 files changed, 325 insertions(+), 132 deletions(-)
diff --git a/src/backend/access/common/printtup.c b/src/backend/access/common/printtup.c
index 2e3eb8f56d3..ab625736ac1 100644
--- a/src/backend/access/common/printtup.c
+++ b/src/backend/access/common/printtup.c
@@ -393,17 +393,19 @@ printtup(TupleTableSlot *slot, DestReceiver *self)
else
{
/* Binary output */
+ Datum outputdatum;
bytea *outputbytes;
- outputbytes = DatumGetByteaP(FunctionCallInvoke(thisState->outstate));
+ outputdatum = FunctionCallInvoke(thisState->outstate);
Assert(!thisState->outstate->isnull);
/*
- * If outputbytes == NULL, the send function directly appended a
+ * If outputdatum == 0, the send function directly appended a
* correctly formatted message.
*/
- if (outputbytes)
+ if (outputdatum)
{
+ outputbytes = DatumGetByteaP(outputdatum);
pq_sendint32(buf, VARSIZE(outputbytes) - VARHDRSZ);
pq_sendbytes(buf, VARDATA(outputbytes),
VARSIZE(outputbytes) - VARHDRSZ);
diff --git a/src/backend/utils/adt/date.c b/src/backend/utils/adt/date.c
index 7f746dd84c9..43284d43fda 100644
--- a/src/backend/utils/adt/date.c
+++ b/src/backend/utils/adt/date.c
@@ -180,7 +180,6 @@ Datum
date_out(PG_FUNCTION_ARGS)
{
DateADT date = PG_GETARG_DATEADT(0);
- char *result;
struct pg_tm tt,
*tm = &tt;
char buf[MAXDATELEN + 1];
@@ -194,8 +193,10 @@ date_out(PG_FUNCTION_ARGS)
EncodeDateOnly(tm, DateStyle, buf);
}
- result = pstrdup(buf);
- PG_RETURN_CSTRING(result);
+ if (pg_send_inout_text(fcinfo, buf, strlen(buf)))
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_CSTRING(pstrdup(buf));
}
/*
@@ -1606,7 +1607,6 @@ Datum
time_out(PG_FUNCTION_ARGS)
{
TimeADT time = PG_GETARG_TIMEADT(0);
- char *result;
struct pg_tm tt,
*tm = &tt;
fsec_t fsec;
@@ -1615,8 +1615,10 @@ time_out(PG_FUNCTION_ARGS)
time2tm(time, tm, &fsec);
EncodeTimeOnly(tm, fsec, false, 0, DateStyle, buf);
- result = pstrdup(buf);
- PG_RETURN_CSTRING(result);
+ if (pg_send_inout_text(fcinfo, buf, strlen(buf)))
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_CSTRING(pstrdup(buf));
}
/*
@@ -1652,11 +1654,21 @@ Datum
time_send(PG_FUNCTION_ARGS)
{
TimeADT time = PG_GETARG_TIMEADT(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendint64(&buf, time);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ pq_sendint64_field(buf, time);
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendint64(&buf, time);
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
Datum
diff --git a/src/backend/utils/adt/float.c b/src/backend/utils/adt/float.c
index 362c29ab803..161d5593f5f 100644
--- a/src/backend/utils/adt/float.c
+++ b/src/backend/utils/adt/float.c
@@ -24,6 +24,7 @@
#include "common/shortest_dec.h"
#include "libpq/pqformat.h"
#include "utils/array.h"
+#include "utils/builtins.h"
#include "utils/float.h"
#include "utils/fmgrprotos.h"
#include "utils/sortsupport.h"
@@ -360,17 +361,18 @@ Datum
float4out(PG_FUNCTION_ARGS)
{
float4 num = PG_GETARG_FLOAT4(0);
- char *ascii = (char *) palloc(32);
+ char ascii[32];
int ndig = FLT_DIG + extra_float_digits;
if (extra_float_digits > 0)
- {
float_to_shortest_decimal_buf(num, ascii);
- PG_RETURN_CSTRING(ascii);
- }
+ else
+ (void) pg_strfromd(ascii, 32, ndig, num);
- (void) pg_strfromd(ascii, 32, ndig, num);
- PG_RETURN_CSTRING(ascii);
+ if (pg_send_inout_text(fcinfo, ascii, strlen(ascii)))
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_CSTRING(pstrdup(ascii));
}
/*
@@ -391,11 +393,21 @@ Datum
float4send(PG_FUNCTION_ARGS)
{
float4 num = PG_GETARG_FLOAT4(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendfloat4(&buf, num);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ pq_sendfloat4_field(buf, num);
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendfloat4(&buf, num);
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
/*
@@ -563,8 +575,18 @@ Datum
float8out(PG_FUNCTION_ARGS)
{
float8 num = PG_GETARG_FLOAT8(0);
+ char ascii[32];
+ int ndig = DBL_DIG + extra_float_digits;
+
+ if (extra_float_digits > 0)
+ double_to_shortest_decimal_buf(num, ascii);
+ else
+ (void) pg_strfromd(ascii, 32, ndig, num);
- PG_RETURN_CSTRING(float8out_internal(num));
+ if (pg_send_inout_text(fcinfo, ascii, strlen(ascii)))
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_CSTRING(pstrdup(ascii));
}
/*
@@ -608,11 +630,21 @@ Datum
float8send(PG_FUNCTION_ARGS)
{
float8 num = PG_GETARG_FLOAT8(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendfloat8(&buf, num);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ pq_sendfloat8_field(buf, num);
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendfloat8(&buf, num);
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
diff --git a/src/backend/utils/adt/int.c b/src/backend/utils/adt/int.c
index de54a43c98d..9a3fda0403a 100644
--- a/src/backend/utils/adt/int.c
+++ b/src/backend/utils/adt/int.c
@@ -74,10 +74,27 @@ Datum
int2out(PG_FUNCTION_ARGS)
{
int16 arg1 = PG_GETARG_INT16(0);
- char *result = (char *) palloc(7); /* sign, 5 digits, '\0' */
+ int maxlen = 7; /* sign, 5 digits, '\0' */
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pg_itoa(arg1, result);
- PG_RETURN_CSTRING(result);
+ if (buf)
+ {
+ int offset;
+ int len;
+
+ len = pg_itoa(arg1, pq_begincountedfield(buf, maxlen, &offset));
+ buf->len += len;
+ pq_endcountedfield(buf, offset);
+
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ char *result = (char *) palloc(maxlen);
+
+ pg_itoa(arg1, result);
+ PG_RETURN_CSTRING(result);
+ }
}
/*
@@ -98,11 +115,21 @@ Datum
int2send(PG_FUNCTION_ARGS)
{
int16 arg1 = PG_GETARG_INT16(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendint16(&buf, arg1);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ pq_sendint16_field(buf, arg1);
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendint16(&buf, arg1);
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
/*
@@ -328,40 +355,22 @@ int4out(PG_FUNCTION_ARGS)
{
int32 arg1 = PG_GETARG_INT32(0);
int maxlen = 12; /* sign, 10 digits, '\0' */
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- if (fcinfo->context && IsA(fcinfo->context, InOutContext))
+ if (buf)
{
- /*
- * Optimized path for output functions called as part of a larger
- * ouput.
- *
- * FIXME: A good chunk of this should obviously be in helper
- * functions.
- */
- InOutContext *inout = castNode(InOutContext, fcinfo->context);
- StringInfo buf = inout->buf;
- int prev_buflen;
+ /* Optimized path for output functions called as part of a larger output. */
+ int offset;
int len;
- uint32 len_net;
-
- /* reserve space for length and the max string length */
- enlargeStringInfo(buf, sizeof(uint32) + maxlen);
-
- /* reserve space for length, to be filled out later */
- prev_buflen = buf->len;
- buf->len += sizeof(uint32);
/*
* Construct string directly in buffer, we don't have to care about
* encoding conversions, because we assume that every encoding
* embodies ascii (XXX: Is that actually true with client encodings?).
*/
- len = pg_ltoa(arg1, buf->data + buf->len);
+ len = pg_ltoa(arg1, pq_begincountedfield(buf, maxlen, &offset));
buf->len += len;
-
- /* update the previously reserved length */
- len_net = pg_hton32(len);
- memcpy(&buf->data[prev_buflen], &len_net, sizeof(uint32));
+ pq_endcountedfield(buf, offset);
PG_RETURN_VOID();
}
@@ -395,16 +404,11 @@ Datum
int4send(PG_FUNCTION_ARGS)
{
int32 arg1 = PG_GETARG_INT32(0);
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- if (fcinfo->context && IsA(fcinfo->context, InOutContext))
+ if (buf)
{
- InOutContext *inout = castNode(InOutContext, fcinfo->context);
-
- /* length of data */
- pq_sendint32(inout->buf, 4);
- /* data itself */
- pq_sendint32(inout->buf, arg1);
-
+ pq_sendint32_field(buf, arg1);
PG_RETURN_VOID();
}
else
diff --git a/src/backend/utils/adt/int8.c b/src/backend/utils/adt/int8.c
index 9b429da86d9..184927f4438 100644
--- a/src/backend/utils/adt/int8.c
+++ b/src/backend/utils/adt/int8.c
@@ -63,19 +63,37 @@ Datum
int8out(PG_FUNCTION_ARGS)
{
int64 val = PG_GETARG_INT64(0);
- char buf[MAXINT8LEN + 1];
- char *result;
- int len;
+ int maxlen = MAXINT8LEN + 1;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- len = pg_lltoa(val, buf) + 1;
+ if (buf)
+ {
+ int offset;
+ int len;
- /*
- * Since the length is already known, we do a manual palloc() and memcpy()
- * to avoid the strlen() call that would otherwise be done in pstrdup().
- */
- result = palloc(len);
- memcpy(result, buf, len);
- PG_RETURN_CSTRING(result);
+ len = pg_lltoa(val, pq_begincountedfield(buf, maxlen, &offset));
+ buf->len += len;
+ pq_endcountedfield(buf, offset);
+
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ char buf[MAXINT8LEN + 1];
+ char *result;
+ int len;
+
+ len = pg_lltoa(val, buf) + 1;
+
+ /*
+ * Since the length is already known, we do a manual palloc() and
+ * memcpy() to avoid the strlen() call that would otherwise be done in
+ * pstrdup().
+ */
+ result = palloc(len);
+ memcpy(result, buf, len);
+ PG_RETURN_CSTRING(result);
+ }
}
/*
diff --git a/src/backend/utils/adt/numeric.c b/src/backend/utils/adt/numeric.c
index cb23dfe9b95..1264a0bb0ff 100644
--- a/src/backend/utils/adt/numeric.c
+++ b/src/backend/utils/adt/numeric.c
@@ -808,11 +808,16 @@ numeric_out(PG_FUNCTION_ARGS)
if (NUMERIC_IS_SPECIAL(num))
{
if (NUMERIC_IS_PINF(num))
- PG_RETURN_CSTRING(pstrdup("Infinity"));
+ str = "Infinity";
else if (NUMERIC_IS_NINF(num))
- PG_RETURN_CSTRING(pstrdup("-Infinity"));
+ str = "-Infinity";
else
- PG_RETURN_CSTRING(pstrdup("NaN"));
+ str = "NaN";
+
+ if (pg_send_inout_text(fcinfo, str, strlen(str)))
+ PG_RETURN_VOID();
+
+ PG_RETURN_CSTRING(pstrdup(str));
}
/*
@@ -822,6 +827,9 @@ numeric_out(PG_FUNCTION_ARGS)
str = get_str_from_var(&x);
+ if (pg_send_inout_text(fcinfo, str, strlen(str)))
+ PG_RETURN_VOID();
+
PG_RETURN_CSTRING(str);
}
@@ -1147,21 +1155,37 @@ numeric_send(PG_FUNCTION_ARGS)
{
Numeric num = PG_GETARG_NUMERIC(0);
NumericVar x;
- StringInfoData buf;
+ StringInfo buf;
+ StringInfoData localbuf;
+ bool inout;
int i;
init_var_from_num(num, &x);
- pq_begintypsend(&buf);
+ buf = pg_get_inout_context_buf(fcinfo);
+ inout = buf != NULL;
+ if (inout)
+ {
+ /* length of data */
+ pq_sendint32(buf, (4 + x.ndigits) * sizeof(int16));
+ }
+ else
+ {
+ pq_begintypsend(&localbuf);
+ buf = &localbuf;
+ }
- pq_sendint16(&buf, x.ndigits);
- pq_sendint16(&buf, x.weight);
- pq_sendint16(&buf, x.sign);
- pq_sendint16(&buf, x.dscale);
+ pq_sendint16(buf, x.ndigits);
+ pq_sendint16(buf, x.weight);
+ pq_sendint16(buf, x.sign);
+ pq_sendint16(buf, x.dscale);
for (i = 0; i < x.ndigits; i++)
- pq_sendint16(&buf, x.digits[i]);
+ pq_sendint16(buf, x.digits[i]);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (inout)
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_BYTEA_P(pq_endtypsend(&localbuf));
}
diff --git a/src/backend/utils/adt/timestamp.c b/src/backend/utils/adt/timestamp.c
index a20e7ea1d11..0c0b7af700a 100644
--- a/src/backend/utils/adt/timestamp.c
+++ b/src/backend/utils/adt/timestamp.c
@@ -227,7 +227,6 @@ Datum
timestamp_out(PG_FUNCTION_ARGS)
{
Timestamp timestamp = PG_GETARG_TIMESTAMP(0);
- char *result;
struct pg_tm tt,
*tm = &tt;
fsec_t fsec;
@@ -242,8 +241,10 @@ timestamp_out(PG_FUNCTION_ARGS)
(errcode(ERRCODE_DATETIME_VALUE_OUT_OF_RANGE),
errmsg("timestamp out of range")));
- result = pstrdup(buf);
- PG_RETURN_CSTRING(result);
+ if (pg_send_inout_text(fcinfo, buf, strlen(buf)))
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_CSTRING(pstrdup(buf));
}
/*
@@ -286,11 +287,21 @@ Datum
timestamp_send(PG_FUNCTION_ARGS)
{
Timestamp timestamp = PG_GETARG_TIMESTAMP(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendint64(&buf, timestamp);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ pq_sendint64_field(buf, timestamp);
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendint64(&buf, timestamp);
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
Datum
@@ -773,7 +784,6 @@ Datum
timestamptz_out(PG_FUNCTION_ARGS)
{
TimestampTz dt = PG_GETARG_TIMESTAMPTZ(0);
- char *result;
int tz;
struct pg_tm tt,
*tm = &tt;
@@ -790,8 +800,10 @@ timestamptz_out(PG_FUNCTION_ARGS)
(errcode(ERRCODE_DATETIME_VALUE_OUT_OF_RANGE),
errmsg("timestamp out of range")));
- result = pstrdup(buf);
- PG_RETURN_CSTRING(result);
+ if (pg_send_inout_text(fcinfo, buf, strlen(buf)))
+ PG_RETURN_VOID();
+ else
+ PG_RETURN_CSTRING(pstrdup(buf));
}
/*
@@ -835,11 +847,21 @@ Datum
timestamptz_send(PG_FUNCTION_ARGS)
{
TimestampTz timestamp = PG_GETARG_TIMESTAMPTZ(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendint64(&buf, timestamp);
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ pq_sendint64_field(buf, timestamp);
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendint64(&buf, timestamp);
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
Datum
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 4948ced7dec..5dac5b39f11 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -289,37 +289,15 @@ Datum
textout(PG_FUNCTION_ARGS)
{
Datum txt = PG_GETARG_DATUM(0);
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- if (fcinfo->context && IsA(fcinfo->context, InOutContext))
+ if (buf)
{
- StringInfo buf = castNode(InOutContext, fcinfo->context)->buf;
text *tunpacked = pg_detoast_datum_packed(DatumGetPointer(txt));
int len = VARSIZE_ANY_EXHDR(tunpacked);
char *data = VARDATA_ANY(tunpacked);
- char *data_converted;
- size_t data_len;
-
- /*
- * Convert text output to the right encoding. For efficiency, this
- * should really happen directly into buf. For that we would have to
- * reserve space for the length first and fill it out after
- * conversion.
- *
- * FIXME: Obviously we would need helpers for this too.
- */
- data_converted = pg_server_to_client(data, len);
-
- if (data == data_converted)
- data_len = len;
- else
- data_len = strlen(data_converted);
-
- /* length */
- pq_sendint32(buf, data_len);
-
- /* actual data */
- appendBinaryStringInfoNT(buf, data_converted, data_len);
+ pq_sendcountedtext(buf, data, len);
if (tunpacked != DatumGetPointer(txt))
pfree(tunpacked);
@@ -356,11 +334,27 @@ Datum
textsend(PG_FUNCTION_ARGS)
{
text *t = PG_GETARG_TEXT_PP(0);
- StringInfoData buf;
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
- pq_begintypsend(&buf);
- pq_sendtext(&buf, VARDATA_ANY(t), VARSIZE_ANY_EXHDR(t));
- PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ if (buf)
+ {
+ int offset;
+
+ /* reserve space for length, to be filled out after conversion */
+ (void) pq_begincountedfield(buf, 0, &offset);
+ pq_sendtext(buf, VARDATA_ANY(t), VARSIZE_ANY_EXHDR(t));
+ pq_endcountedfield(buf, offset);
+
+ PG_RETURN_VOID();
+ }
+ else
+ {
+ StringInfoData buf;
+
+ pq_begintypsend(&buf);
+ pq_sendtext(&buf, VARDATA_ANY(t), VARSIZE_ANY_EXHDR(t));
+ PG_RETURN_BYTEA_P(pq_endtypsend(&buf));
+ }
}
diff --git a/src/include/libpq/pqformat.h b/src/include/libpq/pqformat.h
index bc4ab1381a9..d40f36b5fbc 100644
--- a/src/include/libpq/pqformat.h
+++ b/src/include/libpq/pqformat.h
@@ -94,6 +94,32 @@ pq_writeint64(StringInfoData *pg_restrict buf, uint64 i)
buf->len += sizeof(uint64);
}
+/*
+ * Reserve a length-prefixed field in a StringInfo buffer, returning the
+ * location at which the payload should be written. The payload length is
+ * filled in by pq_endcountedfield().
+ */
+static inline char *
+pq_begincountedfield(StringInfo buf, int maxlen, int *offset)
+{
+ enlargeStringInfo(buf, sizeof(uint32) + maxlen);
+
+ *offset = buf->len;
+ buf->len += sizeof(uint32);
+
+ return buf->data + buf->len;
+}
+
+/* Fill in the length of a field started by pq_begincountedfield(). */
+static inline void
+pq_endcountedfield(StringInfo buf, int offset)
+{
+ int len = buf->len - offset - sizeof(uint32);
+ uint32 len_net = pg_hton32(len);
+
+ memcpy(&buf->data[offset], &len_net, sizeof(uint32));
+}
+
/*
* Append a null-terminated text string (with conversion) to a buffer with
* preallocated space.
@@ -187,6 +213,41 @@ pq_sendint(StringInfo buf, uint32 i, int b)
}
}
+/* append length-prefixed binary fields to a StringInfo buffer */
+static inline void
+pq_sendint16_field(StringInfo buf, uint16 i)
+{
+ pq_sendint32(buf, sizeof(uint16));
+ pq_sendint16(buf, i);
+}
+
+static inline void
+pq_sendint32_field(StringInfo buf, uint32 i)
+{
+ pq_sendint32(buf, sizeof(uint32));
+ pq_sendint32(buf, i);
+}
+
+static inline void
+pq_sendint64_field(StringInfo buf, uint64 i)
+{
+ pq_sendint32(buf, sizeof(uint64));
+ pq_sendint64(buf, i);
+}
+
+static inline void
+pq_sendfloat4_field(StringInfo buf, float4 f)
+{
+ pq_sendint32(buf, sizeof(float4));
+ pq_sendfloat4(buf, f);
+}
+
+static inline void
+pq_sendfloat8_field(StringInfo buf, float8 f)
+{
+ pq_sendint32(buf, sizeof(float8));
+ pq_sendfloat8(buf, f);
+}
extern void pq_begintypsend(StringInfo buf);
extern bytea *pq_endtypsend(StringInfo buf);
diff --git a/src/include/utils/builtins.h b/src/include/utils/builtins.h
index b6a11bfa288..0bf228a746b 100644
--- a/src/include/utils/builtins.h
+++ b/src/include/utils/builtins.h
@@ -15,12 +15,36 @@
#define BUILTINS_H
#include "fmgr.h"
+#include "libpq/pqformat.h"
+#include "nodes/miscnodes.h"
#include "nodes/nodes.h"
#include "utils/fmgrprotos.h"
/* Sign + the most decimal digits an 8-byte number could have */
#define MAXINT8LEN 20
+/* Helpers for output/send functions using InOutContext. */
+static inline StringInfo
+pg_get_inout_context_buf(FunctionCallInfo fcinfo)
+{
+ if (fcinfo->context && IsA(fcinfo->context, InOutContext))
+ return castNode(InOutContext, fcinfo->context)->buf;
+
+ return NULL;
+}
+
+static inline bool
+pg_send_inout_text(FunctionCallInfo fcinfo, const char *str, int slen)
+{
+ StringInfo buf = pg_get_inout_context_buf(fcinfo);
+
+ if (!buf)
+ return false;
+
+ pq_sendcountedtext(buf, str, slen);
+ return true;
+}
+
/* bool.c */
extern bool parse_bool(const char *value, bool *result);
extern bool parse_bool_with_len(const char *value, size_t len, bool *result);
--
2.43.0
--=-=-=
Content-Type: text/x-diff
Content-Disposition: attachment;
filename=v2-0004-Let-pgbench-support-binary-format.patch
^ permalink raw reply [nested|flat] 425+ messages in thread
end of thread, other threads:[~2026-06-02 09:37 UTC | newest]
Thread overview: 425+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2024-10-16 18:24 [PATCH v1 4/5] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-02-20 20:12 [PATCH v2 5/6] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-04-06 14:47 [PATCH v4 6/8] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2025-06-17 12:16 [PATCH v5 07/10] Allow to resize shared memory without restart Dmitrii Dolgov <9erthalion6@gmail.com>
2026-06-02 09:37 [PATCH v2 3/4] Optimize more data type for less memory copy with optional context. Andy Fan <zhihuifan1213@163.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox