Date: Fri, 20 Aug 2021 13:52:10 -0400
From: "alvherre@alvh.no-ip.org" <alvherre@alvh.no-ip.org>
To: "Bossart, Nathan" <bossartn@amazon.com>
Cc: Robert Haas <robertmhaas@gmail.com>,
	Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	"x4mmm@yandex-team.ru" <x4mmm@yandex-team.ru>,
	"a.lubennikova@postgrespro.ru" <a.lubennikova@postgrespro.ru>,
	"hlinnaka@iki.fi" <hlinnaka@iki.fi>,
	"matsumura.ryo@fujitsu.com" <matsumura.ryo@fujitsu.com>,
	"masao.fujii@gmail.com" <masao.fujii@gmail.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: archive status ".ready" files may be created too early
Message-ID: <202108201752.sdsjbgmwee4x@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <72FB62AA-27D2-404B-8A4E-4BA6633AE720@amazon.com>
Archived-At: 
 <https://www.postgresql.org/message-id/202108201752.sdsjbgmwee4x%40alvherre.pgsql>
Precedence: bulk

On 2021-Aug-20, Bossart, Nathan wrote:

> On 8/20/21, 8:29 AM, "Robert Haas" <robertmhaas@gmail.com> wrote:

> > We can't expand the hash table either. It has an initial and maximum
> > size of 16 elements, which means it's basically an expensive array,
> > and which also means that it imposes a new limit of 16 *
> > wal_segment_size on the size of WAL records. If you exceed that limit,
> > I think things just go boom... which I think is not acceptable. I
> > think we can have records in the multi-GB range of wal_level=logical
> > and someone chooses a stupid replica identity setting.
> 
> I was under the impression that shared hash tables could be expanded
> as necessary, but from your note and the following comment, that does
> not seem to be true:

Actually, you were right.  Hash tables in shared memory can be expanded.
There are some limitations (the hash "directory" is fixed size, which
means the hash table get less efficient if it grows too much), but you
can definitely create more hash entries than the initial size.  See for
example element_alloc(), which covers the case of a hash table being
IS_PARTITIONED -- something that only shmem hash tables can be.  Note
that ShmemInitHash passes the HASH_ALLOC flag and uses ShmemAllocNoError
as allocation function, which acquires memory from the shared segment.

This is a minor thing -- it doesn't affect the fact that the hash table
is possibly being misused and inefficient -- but I thought it was worth
pointing out.


As an example, consider the LOCK / PROCLOCK hash tables.  These can
contain more elements than max_backends * max_locks_per_transaction.
Those elements consume shared memory from the "allocation slop" in the
shared memory segment.  It's tough when it happens (as far as I know the
memory is never "returned" once such a hash table grows to use that
space), but it does work.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/