pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: Limit the size of TID lists during parallel GIN build 2+ messages / 1 participants [nested] [flat]
* pgsql: Limit the size of TID lists during parallel GIN build @ 2025-11-04 17:51 Tomas Vondra <tomas.vondra@postgresql.org> 0 siblings, 0 replies; 2+ messages in thread From: Tomas Vondra @ 2025-11-04 17:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Limit the size of TID lists during parallel GIN build When building intermediate TID lists during parallel GIN builds, split the sorted lists into smaller chunks, to limit the amount of memory needed when merging the chunks later. The leader may need to keep in memory up to one chunk per worker, and possibly one extra chunk (before evicting some of the data). The code processing item pointers uses regular palloc/repalloc calls, which means it's subject to the MaxAllocSize (1GB) limit. We could fix this by allowing huge allocations, but that'd require changes in many places without much benefit. Larger chunks do not actually improve performance, so the memory usage would be wasted. Fixed by limiting the chunk size to not hit MaxAllocSize. Each worker gets a fair share. This requires remembering the number of participating workers, in a place that can be accessed from the callback. Luckily, the bs_worker_id field in GinBuildState was unused, so repurpose that. Report by Greg Smith, investigation and fix by me. Batchpatched to 18, where parallel GIN builds were introduced. Reported-by: Gregory Smith <gregsmithpgsql@gmail.com> Discussion: https://postgr.es/m/CAHLJuCWDwn-PE2BMZE4Kux7x5wWt_6RoWtA0mUQffEDLeZ6sfA@mail.gmail.com Backpatch-through: 18 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/c98dffcb7c7010d216dc16d22cb594ef7d65fde1 Modified Files -------------- src/backend/access/gin/gininsert.c | 52 ++++++++++++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 11 deletions(-) ^ permalink raw reply [nested|flat] 2+ messages in thread
* pgsql: Limit the size of TID lists during parallel GIN build @ 2025-11-04 17:51 Tomas Vondra <tomas.vondra@postgresql.org> 0 siblings, 0 replies; 2+ messages in thread From: Tomas Vondra @ 2025-11-04 17:51 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Limit the size of TID lists during parallel GIN build When building intermediate TID lists during parallel GIN builds, split the sorted lists into smaller chunks, to limit the amount of memory needed when merging the chunks later. The leader may need to keep in memory up to one chunk per worker, and possibly one extra chunk (before evicting some of the data). The code processing item pointers uses regular palloc/repalloc calls, which means it's subject to the MaxAllocSize (1GB) limit. We could fix this by allowing huge allocations, but that'd require changes in many places without much benefit. Larger chunks do not actually improve performance, so the memory usage would be wasted. Fixed by limiting the chunk size to not hit MaxAllocSize. Each worker gets a fair share. This requires remembering the number of participating workers, in a place that can be accessed from the callback. Luckily, the bs_worker_id field in GinBuildState was unused, so repurpose that. Report by Greg Smith, investigation and fix by me. Batchpatched to 18, where parallel GIN builds were introduced. Reported-by: Gregory Smith <gregsmithpgsql@gmail.com> Discussion: https://postgr.es/m/CAHLJuCWDwn-PE2BMZE4Kux7x5wWt_6RoWtA0mUQffEDLeZ6sfA@mail.gmail.com Backpatch-through: 18 Branch ------ REL_18_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/a26b753a0894961d82e5f6785ae08959c2755110 Modified Files -------------- src/backend/access/gin/gininsert.c | 52 ++++++++++++++++++++++++++++++-------- 1 file changed, 41 insertions(+), 11 deletions(-) ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2025-11-04 17:51 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-11-04 17:51 pgsql: Limit the size of TID lists during parallel GIN build Tomas Vondra <tomas.vondra@postgresql.org> 2025-11-04 17:51 pgsql: Limit the size of TID lists during parallel GIN build Tomas Vondra <tomas.vondra@postgresql.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox