From: Tomas Vondra <tomas.vondra@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Limit the size of TID lists during parallel GIN build
Date: Tue, 04 Nov 2025 17:51:58 +0000
Message-ID: <E1vGLCI-0051kq-34@gemulon.postgresql.org> (raw)
Limit the size of TID lists during parallel GIN build
When building intermediate TID lists during parallel GIN builds, split
the sorted lists into smaller chunks, to limit the amount of memory
needed when merging the chunks later.
The leader may need to keep in memory up to one chunk per worker, and
possibly one extra chunk (before evicting some of the data). The code
processing item pointers uses regular palloc/repalloc calls, which means
it's subject to the MaxAllocSize (1GB) limit.
We could fix this by allowing huge allocations, but that'd require
changes in many places without much benefit. Larger chunks do not
actually improve performance, so the memory usage would be wasted.
Fixed by limiting the chunk size to not hit MaxAllocSize. Each worker
gets a fair share.
This requires remembering the number of participating workers, in a
place that can be accessed from the callback. Luckily, the bs_worker_id
field in GinBuildState was unused, so repurpose that.
Report by Greg Smith, investigation and fix by me. Batchpatched to 18,
where parallel GIN builds were introduced.
Reported-by: Gregory Smith <gregsmithpgsql@gmail.com>
Discussion: https://postgr.es/m/CAHLJuCWDwn-PE2BMZE4Kux7x5wWt_6RoWtA0mUQffEDLeZ6sfA@mail.gmail.com
Backpatch-through: 18
Branch
------
REL_18_STABLE
Details
-------
https://git.postgresql.org/pg/commitdiff/a26b753a0894961d82e5f6785ae08959c2755110
Modified Files
--------------
src/backend/access/gin/gininsert.c | 52 ++++++++++++++++++++++++++++++--------
1 file changed, 41 insertions(+), 11 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: tomas.vondra@postgresql.org, pgsql-committers@lists.postgresql.org
Subject: Re: pgsql: Limit the size of TID lists during parallel GIN build
In-Reply-To: <E1vGLCI-0051kq-34@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox