public inbox for [email protected]
help / color / mirror / Atom feedFrom: Peter Eisentraut <[email protected]>
To: Tomas Vondra <[email protected]>
To: [email protected]
Subject: Re: pgsql: Allow parallel CREATE INDEX for GIN indexes
Date: Tue, 1 Apr 2025 15:30:48 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
On 07.03.25 22:22, Peter Eisentraut wrote:
> The new tuplesort_getgintuple() in tuplesortvariants.c has a branch that
> does "return false" even though the function's return type is GinTuple
> *. That is probably a mistake. Check please.
>
> Also, this code contains a "pgrminclude ignore", but we don't use those
> anymore.
Fixed committed.
> On 03.03.25 17:10, Tomas Vondra wrote:
>> Allow parallel CREATE INDEX for GIN indexes
>>
>> Allow using parallel workers to build a GIN index, similarly to BTREE
>> and BRIN. For large tables this may result in significant speedup when
>> the build is CPU-bound.
>>
>> The work is divided so that each worker builds index entries on a subset
>> of the table, determined by the regular parallel scan used to read the
>> data. Each worker uses a local tuplesort to sort and merge the entries
>> for the same key. The TID lists do not overlap (for a given key), which
>> means the merge sort simply concatenates the two lists. The merged
>> entries are written into a shared tuplesort for the leader.
>>
>> The leader needs to merge the sorted entries again, before writing them
>> into the index. But this way a significant part of the work happens in
>> the workers, and the leader is left with merging fewer large entries,
>> which is more efficient.
>>
>> Most of the parallelism infrastructure is a simplified copy of the code
>> used by BTREE indexes, omitting the parts irrelevant for GIN indexes
>> (e.g. uniqueness checks).
>>
>> Original patch by me, with reviews and substantial improvements by
>> Matthias van de Meent, certainly enough to make him a co-author.
>>
>> Author: Tomas Vondra, Matthias van de Meent
>> Reviewed-by: Matthias van de Meent, Andy Fan, Kirill Reshke
>> Discussion: https://postgr.es/m/6ab4003f-a8b8-4d75-a67f-
>> f25ad98582dc%40enterprisedb.com
>>
>> Branch
>> ------
>> master
>>
>> Details
>> -------
>> https://git.postgresql.org/pg/
>> commitdiff/8492feb98f6df3f0f03e84ed56f0d1cbb2ac514c
>>
>> Modified Files
>> --------------
>> src/backend/access/gin/gininsert.c | 1649 ++++++++++++++++++++
>> +++++++-
>> src/backend/access/gin/ginutil.c | 30 +-
>> src/backend/access/transam/parallel.c | 4 +
>> src/backend/utils/sort/tuplesortvariants.c | 198 ++++
>> src/include/access/gin.h | 15 +
>> src/include/access/gin_private.h | 1 +
>> src/include/access/gin_tuple.h | 44 +
>> src/include/utils/tuplesort.h | 8 +
>> src/tools/pgindent/typedefs.list | 4 +
>> 9 files changed, 1937 insertions(+), 16 deletions(-)
>>
>
>
>
view thread (4+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: pgsql: Allow parallel CREATE INDEX for GIN indexes
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox