agora inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedFrom: Jack Ng <Jack.Ng@huawei.com>
To: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Ni Ku <jakkuniku@gmail.com>
Subject: RE: Changing shared_buffers without restart
Date: Tue, 6 May 2025 04:23:07 +0000
Message-ID: <fbdc6674ae54420d96368079bf76dd74@huawei.com> (raw)
In-Reply-To: <a2b6laawu4iya4zu737lhgdbvi5bcspxkxqinzogw6o7e2yhke@ay4qtrdlyzhq>
References: <xuuxvlhom2tiinwwnh7r6wds74o2fkwryy6palehytuzm76l4t@3q7lszfqic3b>
<CAExHW5uJujFkJzk_YgvFDKWknBBGoBpjvWB1n+k3szSN-xwN5Q@mail.gmail.com>
<CAExHW5soDB=08jeyqn-W-=+Y86v-a5f_NJRjAdV9_yFrKpJcCw@mail.gmail.com>
<eqs6v4rsboazl67xz3wxc6xjkgrpfybitpl45y3lmb2br67wbj@o7czebb3rlgd>
<CAExHW5sTB1Ow0a1JtVy4PGCPckM7P1CSngJ4+LX3w+yz7rwQfQ@mail.gmail.com>
<ywyt5gecsfjzxlvxbvingd2dps2dedpwq5l4xhfmbcci6xta2e@h7ygoxvep725>
<CAExHW5s1b-_9GFGGOxSVimrt8oe2Y0a7vRhENrd2BSo+0Tkp-A@mail.gmail.com>
<gzb2oqqabzqidlnumrbau2k2fscb5jo6nlpdjnf6pu5y4bk5mo@xoujqgtiqvgr>
<CAExHW5uGozdiDm_BefS7jpQzaisCs4_nLTMF0+S_Pds49pRG_w@mail.gmail.com>
<CAPuPUJwDa_WEVmkknyYHYub5qvBuwKyLLOOiUpyiq6Wd=9FQ_g@mail.gmail.com>
<a2b6laawu4iya4zu737lhgdbvi5bcspxkxqinzogw6o7e2yhke@ay4qtrdlyzhq>
Thanks Dmitry. Right, the coordination mechanism in v4-0006 works as expected in various tests (sorry, I misunderstood some details initially).
I also want to report a couple of minor issues found during testing (which you may be aware of already):
1. For memory segments other the first one ('main'), the start address passed to mmap may not be aligned to 4KB or huge page size (since reserved_offset may not be aligned) and cause mmap to fail.
2. Since the ratio for main/desc/iocv/checkpt/strategy in SHMEM_RESIZE_RATIO are relatively small, I think we need to guard against the case where 'max_available_memory' is too small for the required sizes of these segments (from CalculateShmemSize).
Like when max_available_memory=default and shared_numbers=128kB, 'main' still needs ~109MB, but since only 10% of max_available_memory is reserved for it (~102MB) and start address of the next segment is calculated based on reserved_offset, this would cause the mappings to overlap and memory problems later (I hit this after fixing 1.)
I suppose we can change the minimum value of max_available_memory to be large enough, and may also adjust the ratios in SHMEM_RESIZE_RATIO to ensure the reserved space of those segments are sufficient.
Regards,
Jack Ng
-----Original Message-----
From: Dmitry Dolgov <9erthalion6@gmail.com>
Sent: Monday, April 21, 2025 5:33 AM
To: Ni Ku <jakkuniku@gmail.com>
Cc: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com>; pgsql-hackers@postgresql.org; Robert Haas <robertmhaas@gmail.com>
Subject: Re: Changing shared_buffers without restart
> On Thu, Apr 17, 2025 at 07:05:36PM GMT, Ni Ku wrote:
> I also have a related question about how ftruncate() is used in the patch.
> In my testing I also see that when using ftruncate to shrink a shared
> segment, the memory is freed immediately after the call, even if other
> processes still have that memory mapped, and they will hit SIGBUS if
> they try to access that memory again as the manpage says.
>
> So am I correct to think that, to support the bufferpool shrinking
> case, it would not be safe to call ftruncate in AnonymousShmemResize
> as-is, since at that point other processes may still be using pages
> that belong to the truncated memory?
> It appears that for shrinking we should only call ftruncate when we're
> sure no process will access those pages again (eg, all processes have
> handled the resize interrupt signal barrier). I suppose this can be
> done by the resize coordinator after synchronizing with all the other processes.
> But in that case it seems we cannot use the postmaster as the
> coordinator then? b/c I see some code comments saying the postmaster
> does not have waiting infrastructure... (maybe even if the postmaster
> has waiting infra we don't want to use it anyway since it can be
> blocked for a long time and won't be able to serve other requests).
There is already a coordination infrastructure, implemented in the patch 0006, which will take care of this and prevent access to the shared memory until everything is resized.
view thread (167+ messages) latest in thread
Message-ID: <fbdc6674ae54420d96368079bf76dd74@huawei.com>
Permalink: ../fbdc6674ae54420d96368079bf76dd74@huawei.com/
Also on: postgresql.org/message-id/fbdc6674ae54420d96368079bf76dd74@huawei.com
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: Jack.Ng@huawei.com, 9erthalion6@gmail.com, ashutosh.bapat.oss@gmail.com, robertmhaas@gmail.com, jakkuniku@gmail.com
Subject: RE: Changing shared_buffers without restart
In-Reply-To: <fbdc6674ae54420d96368079bf76dd74@huawei.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox