agora inbox for pgsql-committers@postgresql.orghelp / color / mirror / Atom feed
pgsql: Fix parallel GIN index build with keys larger than 65535 bytes 3+ messages / 1 participants [nested] [flat]
* pgsql: Fix parallel GIN index build with keys larger than 65535 bytes @ 2026-09-14 14:55 Peter Eisentraut <peter@eisentraut.org> 0 siblings, 0 replies; 3+ messages in thread From: Peter Eisentraut @ 2026-09-14 14:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix parallel GIN index build with keys larger than 65535 bytes During a parallel GIN build, each key is serialized into a GinTuple, a transient representation used only while sorting. _gin_build_tuple() lays out the whole tuple (the palloc size, the key memcpy, and the offset of the posting list) from a local variable holding the real key length, but then stored that length in GinTuple.keylen, which was uint16. For a key longer than 65535 bytes, the stored length was thus silently truncated. On read-back, GinTupleGetFirst() and _gin_parse_tuple_items() recompute the posting-list offset from the truncated keylen and land inside the key data, so ginPostingListDecodeAllSegments() decodes garbage: an assertion failure with assertions enabled, and a read past the end of the allocation without, which could in turn lead to a crash or garbage being written into the index. Only parallel builds are affected, because only they materialize a GinTuple; a serial build of the same data succeeds, as index_form_tuple() compresses large keys before the GinMaxItemSize check. To fix, widen GinTuple.keylen to Size, which is what VARSIZE_ANY() returns and what GinBuffer.keylen already uses, and use Size for the local in _gin_build_tuple() too, which was an int that truncated VARSIZE_ANY() the same way. Widening keylen moves GinTuple.data. The key value is accessed in place in data, so data must be MAXALIGN'ed; before, that was only true by accident of the field layout. Make that explicit with alignas(MAXIMUM_ALIGNOF) on data, so that the layout stays correct on 32-bit platforms, where Size is 4 bytes and would otherwise leave data under-aligned when MAXIMUM_ALIGNOF is 8. Bug: #19545 Author: Ewan Young <kdbase.hack@gmail.com> Reported-by: Yuelin Wang <1217816127@qq.com> Reviewed-by: Peter Eisentraut <peter@eisentraut.org> Discussion: https://www.postgresql.org/message-id/flat/19545-0f25b7e47351e8fc%40postgresql.org Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/bad8829872d4d16f0d5de38c2cf9c0e9db86ac22 Modified Files -------------- src/backend/access/gin/gininsert.c | 2 +- src/include/access/gin_tuple.h | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
* pgsql: Fix parallel GIN index build with keys larger than 65535 bytes @ 2026-09-14 14:55 Peter Eisentraut <peter@eisentraut.org> 0 siblings, 0 replies; 3+ messages in thread From: Peter Eisentraut @ 2026-09-14 14:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix parallel GIN index build with keys larger than 65535 bytes During a parallel GIN build, each key is serialized into a GinTuple, a transient representation used only while sorting. _gin_build_tuple() lays out the whole tuple (the palloc size, the key memcpy, and the offset of the posting list) from a local variable holding the real key length, but then stored that length in GinTuple.keylen, which was uint16. For a key longer than 65535 bytes, the stored length was thus silently truncated. On read-back, GinTupleGetFirst() and _gin_parse_tuple_items() recompute the posting-list offset from the truncated keylen and land inside the key data, so ginPostingListDecodeAllSegments() decodes garbage: an assertion failure with assertions enabled, and a read past the end of the allocation without, which could in turn lead to a crash or garbage being written into the index. Only parallel builds are affected, because only they materialize a GinTuple; a serial build of the same data succeeds, as index_form_tuple() compresses large keys before the GinMaxItemSize check. To fix, widen GinTuple.keylen to Size, which is what VARSIZE_ANY() returns and what GinBuffer.keylen already uses, and use Size for the local in _gin_build_tuple() too, which was an int that truncated VARSIZE_ANY() the same way. Widening keylen moves GinTuple.data. The key value is accessed in place in data, so data must be MAXALIGN'ed; before, that was only true by accident of the field layout. Make that explicit with alignas(MAXIMUM_ALIGNOF) on data, so that the layout stays correct on 32-bit platforms, where Size is 4 bytes and would otherwise leave data under-aligned when MAXIMUM_ALIGNOF is 8. Bug: #19545 Author: Ewan Young <kdbase.hack@gmail.com> Reported-by: Yuelin Wang <1217816127@qq.com> Reviewed-by: Peter Eisentraut <peter@eisentraut.org> Discussion: https://www.postgresql.org/message-id/flat/19545-0f25b7e47351e8fc%40postgresql.org Branch ------ REL_19_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/05c4b10fc526a549223d963030057c2e5f709e43 Modified Files -------------- src/backend/access/gin/gininsert.c | 2 +- src/include/access/gin_tuple.h | 9 +++++++-- 2 files changed, 8 insertions(+), 3 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
* pgsql: Fix parallel GIN index build with keys larger than 65535 bytes @ 2026-09-14 14:55 Peter Eisentraut <peter@eisentraut.org> 0 siblings, 0 replies; 3+ messages in thread From: Peter Eisentraut @ 2026-09-14 14:55 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org Fix parallel GIN index build with keys larger than 65535 bytes During a parallel GIN build, each key is serialized into a GinTuple, a transient representation used only while sorting. _gin_build_tuple() lays out the whole tuple (the palloc size, the key memcpy, and the offset of the posting list) from a local variable holding the real key length, but then stored that length in GinTuple.keylen, which was uint16. For a key longer than 65535 bytes, the stored length was thus silently truncated. On read-back, GinTupleGetFirst() and _gin_parse_tuple_items() recompute the posting-list offset from the truncated keylen and land inside the key data, so ginPostingListDecodeAllSegments() decodes garbage: an assertion failure with assertions enabled, and a read past the end of the allocation without, which could in turn lead to a crash or garbage being written into the index. Only parallel builds are affected, because only they materialize a GinTuple; a serial build of the same data succeeds, as index_form_tuple() compresses large keys before the GinMaxItemSize check. To fix, widen GinTuple.keylen to Size, which is what VARSIZE_ANY() returns and what GinBuffer.keylen already uses, and use Size for the local in _gin_build_tuple() too, which was an int that truncated VARSIZE_ANY() the same way. Widening keylen moves GinTuple.data. The key value is accessed in place in data, so data must be MAXALIGN'ed; before, that was only true by accident of the field layout. This branch predates the use of C11 alignas, so force the alignment the way PGAlignedBlock did: put keylen in a union with double and int64 members, which puts data at a MAXALIGN'ed offset on all platforms, including 32-bit ones where Size is 4 bytes and MAXIMUM_ALIGNOF is 8. A static assertion verifies the resulting offset. Bug: #19545 Author: Ewan Young <kdbase.hack@gmail.com> Reported-by: Yuelin Wang <1217816127@qq.com> Reviewed-by: Peter Eisentraut <peter@eisentraut.org> Discussion: https://www.postgresql.org/message-id/flat/19545-0f25b7e47351e8fc%40postgresql.org Branch ------ REL_18_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/937a0e68cb55ac38341f242cf3302906e6e2ec42 Modified Files -------------- src/backend/access/gin/gininsert.c | 12 ++++++------ src/include/access/gin_tuple.h | 25 +++++++++++++++++++++++-- 2 files changed, 29 insertions(+), 8 deletions(-) ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2026-09-14 14:55 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-09-14 14:55 pgsql: Fix parallel GIN index build with keys larger than 65535 bytes Peter Eisentraut <peter@eisentraut.org> 2026-09-14 14:55 pgsql: Fix parallel GIN index build with keys larger than 65535 bytes Peter Eisentraut <peter@eisentraut.org> 2026-09-14 14:55 pgsql: Fix parallel GIN index build with keys larger than 65535 bytes Peter Eisentraut <peter@eisentraut.org>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox