public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v36 4/6] Index skip scan
7+ messages / 4 participants
[nested] [flat]
* [PATCH v36 4/6] Index skip scan
@ 2020-06-08 18:39 Dmitrii Dolgov <[email protected]>
0 siblings, 0 replies; 7+ messages in thread
From: Dmitrii Dolgov @ 2020-06-08 18:39 UTC (permalink / raw)
Implementation of basic Index Skip Scan (see Loose Index Scan in the
wiki [1]) infrastructure on top of IndexOnlyScan and IndexScan.
Introduces a new index am function amskip to allow advance past
duplicate key values in a scan. This innocently looking description
could be a bit tricky on the edge between am specific and common parts
of the implementation, mostly due to different information available at
each level, e.g. visibility. This means the common parts should apply
skipping multiple times if necessary.
Original patch and design were proposed by Thomas Munro [2], revived and
improved by Dmitry Dolgov and Jesper Pedersen.
[1] https://wiki.postgresql.org/wiki/Loose_indexscan
[2] https://www.postgresql.org/message-id/flat/CADLWmXXbTSBxP-MzJuPAYSsL_2f0iPm5VWPbCvDbVvfX93FKkw%40mai...
Author: Jesper Pedersen, Dmitry Dolgov
Reviewed-by: Thomas Munro, David Rowley, Floris Van Nee, Kyotaro Horiguchi, Tomas Vondra, Peter Geoghegan
---
contrib/bloom/blutils.c | 1 +
doc/src/sgml/indexam.sgml | 1 +
src/backend/access/brin/brin.c | 1 +
src/backend/access/gin/ginutil.c | 1 +
src/backend/access/gist/gist.c | 1 +
src/backend/access/hash/hash.c | 1 +
src/backend/access/index/indexam.c | 18 ++
src/backend/access/spgist/spgutils.c | 1 +
src/backend/commands/explain.c | 25 +++
src/backend/executor/nodeIndexonlyscan.c | 97 ++++++++-
src/backend/executor/nodeIndexscan.c | 56 +++++-
src/backend/nodes/copyfuncs.c | 2 +
src/backend/nodes/outfuncs.c | 2 +
src/backend/nodes/readfuncs.c | 2 +
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/path/indxpath.c | 187 +++++++++++++++++-
src/backend/optimizer/path/pathkeys.c | 54 ++++-
src/backend/optimizer/plan/createplan.c | 20 +-
src/backend/optimizer/util/pathnode.c | 37 ++++
src/backend/optimizer/util/plancat.c | 1 +
src/backend/utils/misc/guc.c | 9 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/amapi.h | 8 +
src/include/access/genam.h | 2 +
src/include/access/sdir.h | 7 +
src/include/nodes/execnodes.h | 6 +
src/include/nodes/pathnodes.h | 4 +
src/include/nodes/plannodes.h | 4 +
src/include/optimizer/cost.h | 1 +
src/include/optimizer/pathnode.h | 4 +
src/include/optimizer/paths.h | 5 +-
src/test/regress/expected/sysviews.out | 3 +-
32 files changed, 546 insertions(+), 17 deletions(-)
diff --git a/contrib/bloom/blutils.c b/contrib/bloom/blutils.c
index d3bf8665df..4eec21b99e 100644
--- a/contrib/bloom/blutils.c
+++ b/contrib/bloom/blutils.c
@@ -134,6 +134,7 @@ blhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = blbulkdelete;
amroutine->amvacuumcleanup = blvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = blcostestimate;
amroutine->amoptions = bloptions;
amroutine->amproperty = NULL;
diff --git a/doc/src/sgml/indexam.sgml b/doc/src/sgml/indexam.sgml
index af87f172a7..8c552c22bd 100644
--- a/doc/src/sgml/indexam.sgml
+++ b/doc/src/sgml/indexam.sgml
@@ -150,6 +150,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c
index 7db3ae5ee0..e3e09efa31 100644
--- a/src/backend/access/brin/brin.c
+++ b/src/backend/access/brin/brin.c
@@ -115,6 +115,7 @@ brinhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = brinbulkdelete;
amroutine->amvacuumcleanup = brinvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = brincostestimate;
amroutine->amoptions = brinoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index a400f1fedb..7e83a8ee58 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -66,6 +66,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = ginbulkdelete;
amroutine->amvacuumcleanup = ginvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gincostestimate;
amroutine->amoptions = ginoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 79fe6eb8d6..6fe9f8f5a9 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -87,6 +87,7 @@ gisthandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = gistbulkdelete;
amroutine->amvacuumcleanup = gistvacuumcleanup;
amroutine->amcanreturn = gistcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = gistcostestimate;
amroutine->amoptions = gistoptions;
amroutine->amproperty = gistproperty;
diff --git a/src/backend/access/hash/hash.c b/src/backend/access/hash/hash.c
index 3ec6d528e7..85d44ee58d 100644
--- a/src/backend/access/hash/hash.c
+++ b/src/backend/access/hash/hash.c
@@ -84,6 +84,7 @@ hashhandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = hashbulkdelete;
amroutine->amvacuumcleanup = hashvacuumcleanup;
amroutine->amcanreturn = NULL;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = hashcostestimate;
amroutine->amoptions = hashoptions;
amroutine->amproperty = NULL;
diff --git a/src/backend/access/index/indexam.c b/src/backend/access/index/indexam.c
index 6b9750c244..c7b70997ff 100644
--- a/src/backend/access/index/indexam.c
+++ b/src/backend/access/index/indexam.c
@@ -33,6 +33,7 @@
* index_can_return - does index support index-only scans?
* index_getprocid - get a support procedure OID
* index_getprocinfo - get a support procedure's lookup info
+ * index_skip - advance past duplicate key values in a scan
*
* NOTES
* This file contains the index_ routines which used
@@ -736,6 +737,23 @@ index_can_return(Relation indexRelation, int attno)
return indexRelation->rd_indam->amcanreturn(indexRelation, attno);
}
+/* ----------------
+ * index_skip
+ *
+ * Skip past all tuples where the first 'prefix' columns have the
+ * same value as the last tuple returned in the current scan.
+ * ----------------
+ */
+bool
+index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool scanstart, int prefix)
+{
+ SCAN_CHECKS;
+
+ return scan->indexRelation->rd_indam->amskip(scan, direction,
+ indexdir, scanstart, prefix);
+}
+
/* ----------------
* index_getprocid
*
diff --git a/src/backend/access/spgist/spgutils.c b/src/backend/access/spgist/spgutils.c
index 0efe05e552..0491ea172a 100644
--- a/src/backend/access/spgist/spgutils.c
+++ b/src/backend/access/spgist/spgutils.c
@@ -69,6 +69,7 @@ spghandler(PG_FUNCTION_ARGS)
amroutine->ambulkdelete = spgbulkdelete;
amroutine->amvacuumcleanup = spgvacuumcleanup;
amroutine->amcanreturn = spgcanreturn;
+ amroutine->amskip = NULL;
amroutine->amcostestimate = spgcostestimate;
amroutine->amoptions = spgoptions;
amroutine->amproperty = spgproperty;
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index a283e4d45c..3c6258761a 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -148,6 +148,7 @@ static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainIndentText(ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
+static void ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es);
static void escape_yaml(StringInfo buf, const char *str);
@@ -1096,6 +1097,22 @@ ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
return planstate_tree_walker(planstate, ExplainPreScanNode, rels_used);
}
+/*
+ * ExplainIndexSkipScanKeys -
+ * Append information about index skip scan to es->str.
+ *
+ * Can be used to print the skip prefix size.
+ */
+static void
+ExplainIndexSkipScanKeys(int skipPrefixSize, ExplainState *es)
+{
+ if (skipPrefixSize > 0)
+ {
+ if (es->format != EXPLAIN_FORMAT_TEXT)
+ ExplainPropertyInteger("Distinct Prefix", NULL, skipPrefixSize, es);
+ }
+}
+
/*
* ExplainNode -
* Appends a description of a plan tree to es->str
@@ -1433,6 +1450,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexScan *indexscan = (IndexScan *) plan;
+ ExplainIndexSkipScanKeys(indexscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexscan->indexid,
indexscan->indexorderdir,
es);
@@ -1443,6 +1462,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
{
IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) plan;
+ ExplainIndexSkipScanKeys(indexonlyscan->indexskipprefixsize, es);
+
ExplainIndexScanDetails(indexonlyscan->indexid,
indexonlyscan->indexorderdir,
es);
@@ -1703,6 +1724,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
switch (nodeTag(plan))
{
case T_IndexScan:
+ if (((IndexScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexScan *) plan)->indexqualorig,
"Index Cond", planstate, ancestors, es);
if (((IndexScan *) plan)->indexqualorig)
@@ -1716,6 +1739,8 @@ ExplainNode(PlanState *planstate, List *ancestors,
planstate, es);
break;
case T_IndexOnlyScan:
+ if (((IndexOnlyScan *) plan)->indexskipprefixsize > 0)
+ ExplainPropertyBool("Skip scan", true, es);
show_scan_qual(((IndexOnlyScan *) plan)->indexqual,
"Index Cond", planstate, ancestors, es);
if (((IndexOnlyScan *) plan)->indexqual)
diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c
index 5617ac29e7..c4e4b087a7 100644
--- a/src/backend/executor/nodeIndexonlyscan.c
+++ b/src/backend/executor/nodeIndexonlyscan.c
@@ -41,6 +41,7 @@
#include "miscadmin.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "storage/itemptr.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -62,9 +63,26 @@ IndexOnlyNext(IndexOnlyScanState *node)
EState *estate;
ExprContext *econtext;
ScanDirection direction;
+ ScanDirection readDirection;
IndexScanDesc scandesc;
TupleTableSlot *slot;
ItemPointer tid;
+ ItemPointerData startTid;
+ IndexOnlyScan *indexonlyscan = (IndexOnlyScan *) node->ss.ps.plan;
+
+ /*
+ * Tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
+
+ /*
+ * Index only scan must be aware that in case of skipping we can return to
+ * the starting point due to visibility checks. In this situation we need
+ * to jump further, and number of skipping attempts tell us how far do we
+ * need to do so.
+ */
+ int skipAttempts = 0;
/*
* extract necessary information from index scan node
@@ -72,7 +90,7 @@ IndexOnlyNext(IndexOnlyScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexOnlyScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexonlyscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -114,16 +132,87 @@ IndexOnlyNext(IndexOnlyScanState *node)
node->ioss_OrderByKeys,
node->ioss_NumOrderByKeys);
}
+ else
+ {
+ ItemPointerCopy(&scandesc->xs_heaptid, &startTid);
+ }
+
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->ioss_SkipPrefixSize > 0 &&
+ (node->ioss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexonlyscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted, node->ioss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset ioss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipAttempts = 1;
+ skipped = true;
+ tid = &scandesc->xs_heaptid;
+ }
+ }
+
+ readDirection = skipped ? indexonlyscan->indexorderdir : direction;
/*
* OK, now that we have what we need, fetch the next tuple.
*/
- while ((tid = index_getnext_tid(scandesc, direction)) != NULL)
+ while (skipped || (tid = index_getnext_tid(scandesc, readDirection)) != NULL)
{
bool tuple_from_heap = false;
CHECK_FOR_INTERRUPTS();
+ /*
+ * While doing index only skip scan with advancing and reading in
+ * different directions we can return to the same position where we
+ * started after visibility check. Recognize such situations and skip
+ * more.
+ */
+ if ((readDirection != direction) &&
+ ItemPointerIsValid(&startTid) && ItemPointerEquals(&startTid, tid))
+ {
+ int i;
+ skipAttempts += 1;
+
+ for (i = 0; i < skipAttempts; i++)
+ {
+ if (!index_skip(scandesc, direction,
+ indexonlyscan->indexorderdir,
+ !node->ioss_FirstTupleEmitted,
+ node->ioss_SkipPrefixSize))
+ {
+ node->ioss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ }
+
+ tid = &scandesc->xs_heaptid;
+ }
+
+ skipped = false;
+
/*
* We can skip the heap fetch if the TID references a heap page on
* which all tuples are known visible to everybody. In any case,
@@ -250,6 +339,8 @@ IndexOnlyNext(IndexOnlyScanState *node)
ItemPointerGetBlockNumber(tid),
estate->es_snapshot);
+ node->ioss_FirstTupleEmitted = true;
+
return slot;
}
@@ -504,6 +595,8 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexOnlyScan;
+ indexstate->ioss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->ioss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c
index d0a96a38e0..449aaec3ac 100644
--- a/src/backend/executor/nodeIndexscan.c
+++ b/src/backend/executor/nodeIndexscan.c
@@ -85,6 +85,13 @@ IndexNext(IndexScanState *node)
ScanDirection direction;
IndexScanDesc scandesc;
TupleTableSlot *slot;
+ IndexScan *indexscan = (IndexScan *) node->ss.ps.plan;
+
+ /*
+ * tells if the current position was reached via skipping. In this case
+ * there is no nead for the index_getnext_tid
+ */
+ bool skipped = false;
/*
* extract necessary information from index scan node
@@ -92,7 +99,7 @@ IndexNext(IndexScanState *node)
estate = node->ss.ps.state;
direction = estate->es_direction;
/* flip direction if this is an overall backward scan */
- if (ScanDirectionIsBackward(((IndexScan *) node->ss.ps.plan)->indexorderdir))
+ if (ScanDirectionIsBackward(indexscan->indexorderdir))
{
if (ScanDirectionIsForward(direction))
direction = BackwardScanDirection;
@@ -117,6 +124,12 @@ IndexNext(IndexScanState *node)
node->iss_ScanDesc = scandesc;
+ /* Index skip scan assumes xs_want_itup, so set it to true */
+ if (indexscan->indexskipprefixsize > 0)
+ node->iss_ScanDesc->xs_want_itup = true;
+ else
+ node->iss_ScanDesc->xs_want_itup = false;
+
/*
* If no run-time keys to calculate or they are ready, go ahead and
* pass the scankeys to the index AM.
@@ -127,12 +140,48 @@ IndexNext(IndexScanState *node)
node->iss_OrderByKeys, node->iss_NumOrderByKeys);
}
+ /*
+ * Check if we need to skip to the next key prefix, because we've been
+ * asked to implement DISTINCT.
+ *
+ * When fetching a cursor in the direction opposite to a general scan
+ * direction, the result must be what normal fetching should have
+ * returned, but in reversed order. In other words, return the last or
+ * first scanned tuple in a DISTINCT set, depending on a cursor direction.
+ * Due to that we skip also when the first tuple wasn't emitted yet, but
+ * the directions are opposite.
+ */
+ if (node->iss_SkipPrefixSize > 0 &&
+ (node->iss_FirstTupleEmitted ||
+ ScanDirectionsAreOpposite(direction, indexscan->indexorderdir)))
+ {
+ if (!index_skip(scandesc, direction, indexscan->indexorderdir,
+ !node->iss_FirstTupleEmitted, node->iss_SkipPrefixSize))
+ {
+ /*
+ * Reached end of index. At this point currPos is invalidated, and
+ * we need to reset iss_FirstTupleEmitted, since otherwise after
+ * going backwards, reaching the end of index, and going forward
+ * again we apply skip again. It would be incorrect and lead to an
+ * extra skipped item.
+ */
+ node->iss_FirstTupleEmitted = false;
+ return ExecClearTuple(slot);
+ }
+ else
+ {
+ skipped = true;
+ index_fetch_heap(scandesc, slot);
+ }
+ }
+
/*
* ok, now that we have what we need, fetch the next tuple.
*/
- while (index_getnext_slot(scandesc, direction, slot))
+ while (skipped || index_getnext_slot(scandesc, direction, slot))
{
CHECK_FOR_INTERRUPTS();
+ skipped = false;
/*
* If the index was lossy, we have to recheck the index quals using
@@ -149,6 +198,7 @@ IndexNext(IndexScanState *node)
}
}
+ node->iss_FirstTupleEmitted = true;
return slot;
}
@@ -910,6 +960,8 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags)
indexstate->ss.ps.plan = (Plan *) node;
indexstate->ss.ps.state = estate;
indexstate->ss.ps.ExecProcNode = ExecIndexScan;
+ indexstate->iss_SkipPrefixSize = node->indexskipprefixsize;
+ indexstate->iss_FirstTupleEmitted = false;
/*
* Miscellaneous initialization
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 1f50400fd2..829a1a7c6c 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -493,6 +493,7 @@ _copyIndexScan(const IndexScan *from)
COPY_NODE_FIELD(indexorderbyorig);
COPY_NODE_FIELD(indexorderbyops);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
@@ -518,6 +519,7 @@ _copyIndexOnlyScan(const IndexOnlyScan *from)
COPY_NODE_FIELD(indexorderby);
COPY_NODE_FIELD(indextlist);
COPY_SCALAR_FIELD(indexorderdir);
+ COPY_SCALAR_FIELD(indexskipprefixsize);
return newnode;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index c3a9632992..88254446ce 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -562,6 +562,7 @@ _outIndexScan(StringInfo str, const IndexScan *node)
WRITE_NODE_FIELD(indexorderbyorig);
WRITE_NODE_FIELD(indexorderbyops);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
@@ -576,6 +577,7 @@ _outIndexOnlyScan(StringInfo str, const IndexOnlyScan *node)
WRITE_NODE_FIELD(indexorderby);
WRITE_NODE_FIELD(indextlist);
WRITE_ENUM_FIELD(indexorderdir, ScanDirection);
+ WRITE_INT_FIELD(indexskipprefixsize);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3a18571d0c..9766fc11ed 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -1829,6 +1829,7 @@ _readIndexScan(void)
READ_NODE_FIELD(indexorderbyorig);
READ_NODE_FIELD(indexorderbyops);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
@@ -1848,6 +1849,7 @@ _readIndexOnlyScan(void)
READ_NODE_FIELD(indexorderby);
READ_NODE_FIELD(indextlist);
READ_ENUM_FIELD(indexorderdir, ScanDirection);
+ READ_INT_FIELD(indexskipprefixsize);
READ_DONE();
}
diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 945aa93374..fe6ef62e8f 100644
--- a/src/backend/optimizer/path/costsize.c
+++ b/src/backend/optimizer/path/costsize.c
@@ -125,6 +125,7 @@ int max_parallel_workers_per_gather = 2;
bool enable_seqscan = true;
bool enable_indexscan = true;
bool enable_indexonlyscan = true;
+bool enable_indexskipscan = true;
bool enable_bitmapscan = true;
bool enable_tidscan = true;
bool enable_sort = true;
diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c
index bcb1bc6097..27c945413b 100644
--- a/src/backend/optimizer/path/indxpath.c
+++ b/src/backend/optimizer/path/indxpath.c
@@ -780,6 +780,16 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel,
{
IndexPath *ipath = (IndexPath *) lfirst(lc);
+ /*
+ * To prevent unique paths from index skip scans being potentially used
+ * when not needed scan keep them in a separate pathlist.
+ */
+ if (ipath->indexskipprefix != 0)
+ {
+ add_unique_path(rel, (Path *) ipath);
+ continue;
+ }
+
if (index->amhasgettuple)
add_path(rel, (Path *) ipath);
@@ -862,12 +872,15 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
double loop_count;
List *orderbyclauses;
List *orderbyclausecols;
- List *index_pathkeys;
+ List *index_pathkeys = NIL;
List *useful_pathkeys;
+ List *index_pathkeys_pos = NIL;
bool found_lower_saop_clause;
bool pathkeys_possibly_useful;
bool index_is_ordered;
bool index_only_scan;
+ bool not_empty_qual = false;
+ bool can_skip;
int indexcol;
/*
@@ -985,7 +998,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- ForwardScanDirection);
+ ForwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
orderbyclauses = NIL;
@@ -1017,6 +1031,120 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
index_only_scan = (scantype != ST_BITMAPSCAN &&
check_index_only(rel, index));
+ /* Check if an index skip scan is possible. */
+ can_skip = enable_indexskipscan & index->amcanskip;
+
+ if (can_skip)
+ {
+ /*
+ * Skip scan is not supported when there are qual conditions, which are not
+ * covered by index. The reason for that is that those conditions are
+ * evaluated later, already after skipping was applied.
+ *
+ * TODO: This implementation is too restrictive, and doesn't allow e.g.
+ * index expressions. For that we need to examine index_clauses too.
+ */
+ if (root->parse->jointree != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, (List *)root->parse->jointree->quals)
+ {
+ Node *expr, *qual = (Node *) lfirst(lc);
+ Var *var;
+ bool found = false;
+
+ if (!is_opclause(qual))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ expr = get_leftop(qual);
+
+ if (!IsA(expr, Var))
+ {
+ not_empty_qual = true;
+ break;
+ }
+
+ var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found)
+ {
+ not_empty_qual = true;
+ break;
+ }
+ }
+ }
+
+ /*
+ * For an index scan verify that index fully covers distinct
+ * expressions, otherwise there is not enough information for skipping
+ */
+ if (!index_only_scan && root->query_uniquekeys != NULL)
+ {
+ ListCell *lc;
+
+ foreach(lc, root->query_uniquekeys)
+ {
+ UniqueKey *uniqueKey = (UniqueKey *) lfirst(lc);
+ ListCell *lc1;
+
+ foreach(lc1, uniqueKey->exprs)
+ {
+ Expr *expr = (Expr *) lfirst(lc1);
+ bool found = false;
+
+ if (!IsA(expr, Var))
+ {
+ ListCell *lc2;
+
+ foreach(lc2, index->indexprs)
+ {
+ if(equal(lfirst(lc1), lfirst(lc2)))
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+ else
+ {
+ Var *var = (Var *) expr;
+
+ for (int i = 0; i < index->ncolumns; i++)
+ {
+ if (index->indexkeys[i] == var->varattno)
+ {
+ found = true;
+ break;
+ }
+ }
+ }
+
+ if (!found)
+ {
+ can_skip = false;
+ break;
+ }
+ }
+
+ if (!can_skip)
+ break;
+ }
+ }
+ }
+
/*
* 4. Generate an indexscan path if there are relevant restriction clauses
* in the current clauses, OR the index ordering is potentially useful for
@@ -1040,6 +1168,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/*
* If appropriate, consider parallel index scan. We don't allow
* parallel index scan for bitmap index scans.
@@ -1078,7 +1232,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
if (index_is_ordered && pathkeys_possibly_useful)
{
index_pathkeys = build_index_pathkeys(root, index,
- BackwardScanDirection);
+ BackwardScanDirection,
+ &index_pathkeys_pos);
useful_pathkeys = truncate_useless_pathkeys(root, rel,
index_pathkeys);
if (useful_pathkeys != NIL)
@@ -1095,6 +1250,32 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel,
false);
result = lappend(result, ipath);
+ /* Consider index skip scan as well */
+ if (root->query_uniquekeys != NULL && can_skip && !not_empty_qual)
+ {
+ int numusefulkeys = list_length(useful_pathkeys);
+ int numsortkeys = list_length(root->query_pathkeys);
+
+ if (numusefulkeys == numsortkeys)
+ {
+ int prefix;
+ if (list_length(root->distinct_pathkeys) > 0)
+ prefix = find_index_prefix_for_pathkey(index_pathkeys,
+ index_pathkeys_pos,
+ llast_node(PathKey,
+ root->distinct_pathkeys));
+ else
+ /* all are distinct keys are constant and optimized away.
+ * skipping with 1 is sufficient as all are constant anyway
+ */
+ prefix = 1;
+
+ result = lappend(result,
+ create_skipscan_unique_path(root, index,
+ (Path *) ipath, prefix));
+ }
+ }
+
/* If appropriate, consider parallel index scan */
if (index->amcanparallel &&
rel->consider_parallel && outer_relids == NULL &&
diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c
index 97e1c2de83..61c0ea98e7 100644
--- a/src/backend/optimizer/path/pathkeys.c
+++ b/src/backend/optimizer/path/pathkeys.c
@@ -522,6 +522,47 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* NEW PATHKEY FORMATION
****************************************************************************/
+/*
+ * Find the prefix size for a specific path key in an index. For example, an
+ * index with (a,b,c) finding path key b will return prefix 2. Optionally
+ * pathkeys_positions can be provided, to specify at which position in the
+ * original pathkey list this particular key could be found (this is helpful
+ * when we deal with redundant pathkeys).
+ *
+ * Returns 0 when not found.
+ */
+int
+find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkeys_positions,
+ PathKey *target_pathkey)
+{
+ ListCell *lc;
+ int i;
+
+ i = 0;
+ foreach(lc, index_pathkeys)
+ {
+ PathKey *cpathkey = (PathKey *) lfirst(lc);
+
+ if (cpathkey == target_pathkey)
+ {
+ /*
+ * Prefix expected to start from 1, increment positions since
+ * they're 0 based.
+ */
+ if (pathkeys_positions != NIL &&
+ pathkeys_positions->length > i)
+ return list_nth_int(pathkeys_positions, i) + 1;
+ else
+ return i + 1;
+ }
+
+ i++;
+ }
+
+ return 0;
+}
+
/*
* build_index_pathkeys
* Build a pathkeys list that describes the ordering induced by an index
@@ -534,7 +575,9 @@ get_cheapest_parallel_safe_total_inner(List *paths)
* We iterate only key columns of covering indexes, since non-key columns
* don't influence index ordering. The result is canonical, meaning that
* redundant pathkeys are removed; it may therefore have fewer entries than
- * there are key columns in the index.
+ * there are key columns in the index. Since by removing redundant pathkeys the
+ * information about original position is lost, return it via positions
+ * argument.
*
* Another reason for stopping early is that we may be able to tell that
* an index column's sort order is uninteresting for this query. However,
@@ -545,7 +588,8 @@ get_cheapest_parallel_safe_total_inner(List *paths)
List *
build_index_pathkeys(PlannerInfo *root,
IndexOptInfo *index,
- ScanDirection scandir)
+ ScanDirection scandir,
+ List **positions)
{
List *retval = NIL;
ListCell *lc;
@@ -554,6 +598,8 @@ build_index_pathkeys(PlannerInfo *root,
if (index->sortopfamily == NULL)
return NIL; /* non-orderable index */
+ *positions = NIL;
+
i = 0;
foreach(lc, index->indextlist)
{
@@ -607,7 +653,11 @@ build_index_pathkeys(PlannerInfo *root,
* for this query. Add it to list, unless it's redundant.
*/
if (!pathkey_is_redundant(cpathkey, retval))
+ {
retval = lappend(retval, cpathkey);
+ *positions = lappend_int(*positions,
+ foreach_current_index(lc));
+ }
}
else
{
diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c
index 99278eed93..7185424c20 100644
--- a/src/backend/optimizer/plan/createplan.c
+++ b/src/backend/optimizer/plan/createplan.c
@@ -177,12 +177,14 @@ static IndexScan *make_indexscan(List *qptlist, List *qpqual, Index scanrelid,
Oid indexid, List *indexqual, List *indexqualorig,
List *indexorderby, List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static IndexOnlyScan *make_indexonlyscan(List *qptlist, List *qpqual,
Index scanrelid, Oid indexid,
List *indexqual, List *indexorderby,
List *indextlist,
- ScanDirection indexscandir);
+ ScanDirection indexscandir,
+ int skipprefix);
static BitmapIndexScan *make_bitmap_indexscan(Index scanrelid, Oid indexid,
List *indexqual,
List *indexqualorig);
@@ -2987,7 +2989,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexquals,
fixed_indexorderbys,
best_path->indexinfo->indextlist,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
else
scan_plan = (Scan *) make_indexscan(tlist,
qpqual,
@@ -2998,7 +3001,8 @@ create_indexscan_plan(PlannerInfo *root,
fixed_indexorderbys,
indexorderbys,
indexorderbyops,
- best_path->indexscandir);
+ best_path->indexscandir,
+ best_path->indexskipprefix);
copy_generic_path_info(&scan_plan->plan, &best_path->path);
@@ -5265,7 +5269,8 @@ make_indexscan(List *qptlist,
List *indexorderby,
List *indexorderbyorig,
List *indexorderbyops,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexScan *node = makeNode(IndexScan);
Plan *plan = &node->scan.plan;
@@ -5282,6 +5287,7 @@ make_indexscan(List *qptlist,
node->indexorderbyorig = indexorderbyorig;
node->indexorderbyops = indexorderbyops;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
@@ -5294,7 +5300,8 @@ make_indexonlyscan(List *qptlist,
List *indexqual,
List *indexorderby,
List *indextlist,
- ScanDirection indexscandir)
+ ScanDirection indexscandir,
+ int skipPrefixSize)
{
IndexOnlyScan *node = makeNode(IndexOnlyScan);
Plan *plan = &node->scan.plan;
@@ -5309,6 +5316,7 @@ make_indexonlyscan(List *qptlist,
node->indexorderby = indexorderby;
node->indextlist = indextlist;
node->indexorderdir = indexscandir;
+ node->indexskipprefixsize = skipPrefixSize;
return node;
}
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index d4abb3cb47..e63ba85efd 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -3003,6 +3003,43 @@ create_upper_unique_path(PlannerInfo *root,
return pathnode;
}
+/*
+ * create_skipscan_unique_path
+ * Creates a pathnode the same as an existing IndexPath except based on
+ * skipping duplicate values. This may or may not be cheaper than using
+ * create_upper_unique_path.
+ *
+ * The input path must be an IndexPath for an index that supports amskip.
+ */
+IndexPath *
+create_skipscan_unique_path(PlannerInfo *root, IndexOptInfo *index,
+ Path *basepath, int prefix)
+{
+ IndexPath *pathnode = makeNode(IndexPath);
+ int numDistinctRows;
+ UniqueKey *ukey;
+
+ Assert(IsA(basepath, IndexPath));
+
+ /* We don't want to modify basepath, so make a copy. */
+ memcpy(pathnode, basepath, sizeof(IndexPath));
+
+ ukey = linitial_node(UniqueKey, root->query_uniquekeys);
+
+ Assert(prefix > 0);
+ pathnode->indexskipprefix = prefix;
+ pathnode->path.uniquekeys = root->query_uniquekeys;
+
+ numDistinctRows = estimate_num_groups(root, ukey->exprs,
+ pathnode->path.rows,
+ NULL);
+
+ pathnode->path.total_cost = pathnode->path.startup_cost * numDistinctRows;
+ pathnode->path.rows = numDistinctRows;
+
+ return pathnode;
+}
+
/*
* create_agg_path
* Creates a pathnode that represents performing aggregation/grouping
diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c
index 0b2f9d398a..6da71e5828 100644
--- a/src/backend/optimizer/util/plancat.c
+++ b/src/backend/optimizer/util/plancat.c
@@ -272,6 +272,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent,
info->amoptionalkey = amroutine->amoptionalkey;
info->amsearcharray = amroutine->amsearcharray;
info->amsearchnulls = amroutine->amsearchnulls;
+ info->amcanskip = (amroutine->amskip != NULL);
info->amcanparallel = amroutine->amcanparallel;
info->amhasgettuple = (amroutine->amgettuple != NULL);
info->amhasgetbitmap = amroutine->amgetbitmap != NULL &&
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 6f603cbbe8..b04ef1811d 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -957,6 +957,15 @@ static struct config_bool ConfigureNamesBool[] =
true,
NULL, NULL, NULL
},
+ {
+ {"enable_indexskipscan", PGC_USERSET, QUERY_TUNING_METHOD,
+ gettext_noop("Enables the planner's use of index-skip-scan plans."),
+ NULL
+ },
+ &enable_indexskipscan,
+ true,
+ NULL, NULL, NULL
+ },
{
{"enable_bitmapscan", PGC_USERSET, QUERY_TUNING_METHOD,
gettext_noop("Enables the planner's use of bitmap-scan plans."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5a0b8e9821..947dff83a5 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -357,6 +357,7 @@
#enable_hashjoin = on
#enable_indexscan = on
#enable_indexonlyscan = on
+#enable_indexskipscan = on
#enable_material = on
#enable_mergejoin = on
#enable_nestloop = on
diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h
index 4325faa460..85cca4d520 100644
--- a/src/include/access/amapi.h
+++ b/src/include/access/amapi.h
@@ -130,6 +130,13 @@ typedef void (*amrescan_function) (IndexScanDesc scan,
typedef bool (*amgettuple_function) (IndexScanDesc scan,
ScanDirection direction);
+/* skip past duplicates in a given prefix */
+typedef bool (*amskip_function) (IndexScanDesc scan,
+ ScanDirection dir,
+ ScanDirection indexdir,
+ bool start,
+ int prefix);
+
/* fetch all valid tuples */
typedef int64 (*amgetbitmap_function) (IndexScanDesc scan,
TIDBitmap *tbm);
@@ -231,6 +238,7 @@ typedef struct IndexAmRoutine
amendscan_function amendscan;
ammarkpos_function ammarkpos; /* can be NULL */
amrestrpos_function amrestrpos; /* can be NULL */
+ amskip_function amskip; /* can be NULL */
/* interface functions to support parallel index scans */
amestimateparallelscan_function amestimateparallelscan; /* can be NULL */
diff --git a/src/include/access/genam.h b/src/include/access/genam.h
index 931257bd81..95ce4a5d80 100644
--- a/src/include/access/genam.h
+++ b/src/include/access/genam.h
@@ -180,6 +180,8 @@ extern IndexBulkDeleteResult *index_bulk_delete(IndexVacuumInfo *info,
extern IndexBulkDeleteResult *index_vacuum_cleanup(IndexVacuumInfo *info,
IndexBulkDeleteResult *stats);
extern bool index_can_return(Relation indexRelation, int attno);
+extern bool index_skip(IndexScanDesc scan, ScanDirection direction,
+ ScanDirection indexdir, bool start, int prefix);
extern RegProcedure index_getprocid(Relation irel, AttrNumber attnum,
uint16 procnum);
extern FmgrInfo *index_getprocinfo(Relation irel, AttrNumber attnum,
diff --git a/src/include/access/sdir.h b/src/include/access/sdir.h
index 23feb90986..094a127464 100644
--- a/src/include/access/sdir.h
+++ b/src/include/access/sdir.h
@@ -55,4 +55,11 @@ typedef enum ScanDirection
#define ScanDirectionIsForward(direction) \
((bool) ((direction) == ForwardScanDirection))
+/*
+ * ScanDirectionsAreOpposite
+ * True iff scan directions are backward/forward or forward/backward.
+ */
+#define ScanDirectionsAreOpposite(dirA, dirB) \
+ ((bool) (dirA != NoMovementScanDirection && dirA == -dirB))
+
#endif /* SDIR_H */
diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h
index 6f96b31fb4..68563b1d1b 100644
--- a/src/include/nodes/execnodes.h
+++ b/src/include/nodes/execnodes.h
@@ -1429,6 +1429,8 @@ typedef struct IndexScanState
ExprContext *iss_RuntimeContext;
Relation iss_RelationDesc;
struct IndexScanDescData *iss_ScanDesc;
+ int iss_SkipPrefixSize;
+ bool iss_FirstTupleEmitted;
/* These are needed for re-checking ORDER BY expr ordering */
pairingheap *iss_ReorderQueue;
@@ -1458,6 +1460,8 @@ typedef struct IndexScanState
* TableSlot slot for holding tuples fetched from the table
* VMBuffer buffer in use for visibility map testing, if any
* PscanLen size of parallel index-only scan descriptor
+ * SkipPrefixSize number of keys for skip-based DISTINCT
+ * FirstTupleEmitted has the first tuple been emitted
* ----------------
*/
typedef struct IndexOnlyScanState
@@ -1476,6 +1480,8 @@ typedef struct IndexOnlyScanState
struct IndexScanDescData *ioss_ScanDesc;
TupleTableSlot *ioss_TableSlot;
Buffer ioss_VMBuffer;
+ int ioss_SkipPrefixSize;
+ bool ioss_FirstTupleEmitted;
Size ioss_PscanLen;
} IndexOnlyScanState;
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index a5c406bd4e..7287594582 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -1229,6 +1229,9 @@ typedef struct Path
* we need not recompute them when considering using the same index in a
* bitmap index/heap scan (see BitmapHeapPath). The costs of the IndexPath
* itself represent the costs of an IndexScan or IndexOnlyScan plan type.
+ *
+ * 'indexskipprefix' represents the number of columns to consider for skip
+ * scans.
*----------
*/
typedef struct IndexPath
@@ -1241,6 +1244,7 @@ typedef struct IndexPath
ScanDirection indexscandir;
Cost indextotalcost;
Selectivity indexselectivity;
+ int indexskipprefix;
} IndexPath;
/*
diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h
index 83e01074ed..efafdf9361 100644
--- a/src/include/nodes/plannodes.h
+++ b/src/include/nodes/plannodes.h
@@ -409,6 +409,8 @@ typedef struct IndexScan
List *indexorderbyorig; /* the same in original form */
List *indexorderbyops; /* OIDs of sort ops for ORDER BY exprs */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexScan;
/* ----------------
@@ -436,6 +438,8 @@ typedef struct IndexOnlyScan
List *indexorderby; /* list of index ORDER BY exprs */
List *indextlist; /* TargetEntry list describing index's cols */
ScanDirection indexorderdir; /* forward or backward or don't care */
+ int indexskipprefixsize; /* the size of the prefix for distinct
+ * scans */
} IndexOnlyScan;
/* ----------------
diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h
index 613db8eab6..c0f176eaaa 100644
--- a/src/include/optimizer/cost.h
+++ b/src/include/optimizer/cost.h
@@ -50,6 +50,7 @@ extern PGDLLIMPORT int max_parallel_workers_per_gather;
extern PGDLLIMPORT bool enable_seqscan;
extern PGDLLIMPORT bool enable_indexscan;
extern PGDLLIMPORT bool enable_indexonlyscan;
+extern PGDLLIMPORT bool enable_indexskipscan;
extern PGDLLIMPORT bool enable_bitmapscan;
extern PGDLLIMPORT bool enable_tidscan;
extern PGDLLIMPORT bool enable_sort;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index 6796ad8cb7..8ec1780a56 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -207,6 +207,10 @@ extern UpperUniquePath *create_upper_unique_path(PlannerInfo *root,
Path *subpath,
int numCols,
double numGroups);
+extern IndexPath *create_skipscan_unique_path(PlannerInfo *root,
+ IndexOptInfo *index,
+ Path *subpath,
+ int prefix);
extern AggPath *create_agg_path(PlannerInfo *root,
RelOptInfo *rel,
Path *subpath,
diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h
index 0cb8030e33..6ebcb0e0a4 100644
--- a/src/include/optimizer/paths.h
+++ b/src/include/optimizer/paths.h
@@ -198,8 +198,11 @@ extern Path *get_cheapest_fractional_path_for_pathkeys(List *paths,
Relids required_outer,
double fraction);
extern Path *get_cheapest_parallel_safe_total_inner(List *paths);
+extern int find_index_prefix_for_pathkey(List *index_pathkeys,
+ List *pathkey_positions,
+ PathKey *target_pathkey);
extern List *build_index_pathkeys(PlannerInfo *root, IndexOptInfo *index,
- ScanDirection scandir);
+ ScanDirection scandir, List **positions);
extern List *build_partition_pathkeys(PlannerInfo *root, RelOptInfo *partrel,
ScanDirection scandir, bool *partialkeys);
extern List *build_expression_pathkey(PlannerInfo *root, Expr *expr,
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 06c4c3e476..ce883e5060 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -79,6 +79,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_incremental_sort | on
enable_indexonlyscan | on
enable_indexscan | on
+ enable_indexskipscan | on
enable_material | on
enable_mergejoin | on
enable_nestloop | on
@@ -90,7 +91,7 @@ select name, setting from pg_settings where name like 'enable%';
enable_seqscan | on
enable_sort | on
enable_tidscan | on
-(18 rows)
+(21 rows)
-- Test that the pg_timezone_names and pg_timezone_abbrevs views are
-- more-or-less working. We can't test their contents in any great detail
--
2.21.0
--op7mf2i72xk5yu7c
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v36-0005-Btree-implementation-of-skipping.patch"
^ permalink raw reply [nested|flat] 7+ messages in thread
* Parallel CREATE INDEX for GIN indexes
@ 2024-05-02 15:19 Tomas Vondra <[email protected]>
0 siblings, 2 replies; 7+ messages in thread
From: Tomas Vondra @ 2024-05-02 15:19 UTC (permalink / raw)
To: PostgreSQL Hackers <[email protected]>
Hi,
In PG17 we shall have parallel CREATE INDEX for BRIN indexes, and back
when working on that I was thinking how difficult would it be to do
something similar to do that for other index types, like GIN. I even had
that on my list of ideas to pitch to potential contributors, as I was
fairly sure it's doable and reasonably isolated / well-defined.
However, I was not aware of any takers, so a couple days ago on a slow
weekend I took a stab at it. And yes, it's doable - attached is a fairly
complete, tested and polished version of the feature, I think. It turned
out to be a bit more complex than I expected, for reasons that I'll get
into when discussing the patches.
First, let's talk about the benefits - how much faster is that than the
single-process build we have for GIN indexes? I do have a table with the
archive of all our mailing lists - it's ~1.5M messages, table is ~21GB
(raw dump is about 28GB). This does include simple text data (message
body), JSONB (headers) and tsvector (full-text on message body).
If I do CREATE index with different number of workers (0 means serial
build), I get this timings (in seconds):
workers trgm tsvector jsonb jsonb (hash)
-----------------------------------------------------
0 1240 378 104 57
1 773 196 59 85
2 548 163 51 78
3 423 153 45 75
4 362 142 43 75
5 323 134 40 70
6 295 130 39 73
Perhaps an easier to understand result is this table with relative
timing compared to serial build:
workers trgm tsvector jsonb jsonb (hash)
-----------------------------------------------------
1 62% 52% 57% 149%
2 44% 43% 49% 136%
3 34% 40% 43% 132%
4 29% 38% 41% 131%
5 26% 35% 39% 123%
6 24% 34% 38% 129%
This shows the benefits are pretty nice, depending on the opclass. For
most indexes it's maybe ~3-4x faster, which is nice, and I don't think
it's possible to do much better - the actual index inserts can happen
from a single process only, which is the main limit.
For some of the opclasses it can regress (like the jsonb_path_ops). I
don't think that's a major issue. Or more precisely, I'm not surprised
by it. It'd be nice to be able to disable the parallel builds in these
cases somehow, but I haven't thought about that.
I do plan to do some tests with btree_gin, but I don't expect that to
behave significantly differently.
There are small variations in the index size, when built in the serial
way and the parallel way. It's generally within ~5-10%, and I believe
it's due to the serial build adding the TIDs incrementally, while the
build adds them in much larger chunks (possibly even in one chunk with
all the TIDs for the key). I believe the same size variation can happen
if the index gets built in a different way, e.g. by inserting the data
in a different order, etc. I did a number of tests to check if the index
produces the correct results, and I haven't found any issues. So I think
this is OK, and neither a problem nor an advantage of the patch.
Now, let's talk about the code - the series has 7 patches, with 6
non-trivial parts doing changes in focused and easier to understand
pieces (I hope so).
1) v20240502-0001-Allow-parallel-create-for-GIN-indexes.patch
This is the initial feature, adding the "basic" version, implemented as
pretty much 1:1 copy of the BRIN parallel build and minimal changes to
make it work for GIN (mostly about how to store intermediate results).
The basic idea is that the workers do the regular build, but instead of
flushing the data into the index after hitting the memory limit, it gets
written into a shared tuplesort and sorted by the index key. And the
leader then reads this sorted data, accumulates the TID for a given key
and inserts that into the index in one go.
2) v20240502-0002-Use-mergesort-in-the-leader-process.patch
The approach implemented by 0001 works, but there's a little bit of
issue - if there are many distinct keys (e.g. for trigrams that can
happen very easily), the workers will hit the memory limit with only
very short TID lists for most keys. For serial build that means merging
the data into a lot of random places, and in parallel build it means the
leader will have to merge a lot of tiny lists from many sorted rows.
Which can be quite annoying and expensive, because the leader does so
using qsort() in the serial part. It'd be better to ensure most of the
sorting happens in the workers, and the leader can do a mergesort. But
the mergesort must not happen too often - merging many small lists is
not cheaper than a single qsort (especially when the lists overlap).
So this patch changes the workers to process the data in two phases. The
first works as before, but the data is flushed into a local tuplesort.
And then each workers sorts the results it produced, and combines them
into results with much larger TID lists, and those results are written
to the shared tuplesort. So the leader only gets very few lists to
combine for a given key - usually just one list per worker.
3) v20240502-0003-Remove-the-explicit-pg_qsort-in-workers.patch
In 0002 the workers still do an explicit qsort() on the TID list before
writing the data into the shared tuplesort. But we can do better - the
workers can do a merge sort too. To help with this, we add the first TID
to the tuplesort tuple, and sort by that too - it helps the workers to
process the data in an order that allows simple concatenation instead of
the full mergesort.
Note: There's a non-obvious issue due to parallel scans always being
"sync scans", which may lead to very "wide" TID ranges when the scan
wraps around. More about that later.
4) v20240502-0004-Compress-TID-lists-before-writing-tuples-t.patch
The parallel build passes data between processes using temporary files,
which means it may need significant amount of disk space. For BRIN this
was not a major concern, because the summaries tend to be pretty small.
But for GIN that's not the case, and the two-phase processing introduced
by 0002 make it worse, because the worker essentially creates another
copy of the intermediate data. It does not need to copy the key, so
maybe it's not exactly 2x the space requirement, but in the worst case
it's not far from that.
But there's a simple way how to improve this - the TID lists tend to be
very compressible, and GIN already implements a very light-weight TID
compression, so this patch does just that - when building the tuple to
be written into the tuplesort, we just compress the TIDs.
5) v20240502-0005-Collect-and-print-compression-stats.patch
This patch simply collects some statistics about the compression, to
show how much it reduces the amounts of data in the various phases. The
data I've seen so far usually show ~75% compression in the first phase,
and ~30% compression in the second phase.
That is, in the first phase we save ~25% of space, in the second phase
we save ~70% of space. An example of the log messages from this patch,
for one worker (of two) in the trigram phase says:
LOG: _gin_parallel_scan_and_build raw 10158870494 compressed 7519211584
ratio 74.02%
LOG: _gin_process_worker_data raw 4593563782 compressed 1314800758
ratio 28.62%
Put differently, a single-phase version without compression (as in 0001)
would need ~10GB of disk space per worker. With compression, we need
only about ~8.8GB for both phases (or ~7.5GB for the first phase alone).
I do think these numbers look pretty good. The numbers are different for
other opclasses (trigrams are rather extreme in how much space they
need), but the overall behavior is the same.
6) v20240502-0006-Enforce-memory-limit-when-combining-tuples.patch
Until this part, there's no limit on memory used by combining results
for a single index key - it'll simply use as much memory as needed to
combine all the TID lists. Which may not be a huge issue because each
TID is only 6B, and we can accumulate a lot of those in a couple MB. And
a parallel CREATE INDEX usually runs with a fairly significant values of
maintenance_work_mem (in fact it requires it to even allow parallel).
But still, there should be some memory limit.
It however is not as simple as dumping current state into the index,
because the TID lists produced by the workers may overlap, so the tail
of the list may still receive TIDs from some future TID list. And that's
a problem because ginEntryInsert() expects to receive TIDs in order, and
if that's not the case it may fail with "could not split GIN page".
But we already have the first TID for each sort tuple (and we consider
it when sorting the data), and this is useful for deducing how far we
can flush the data, and keep just the minimal part of the TID list that
may change by merging.
So this patch implements that - it introduces the concept of "freezing"
the head of the TID list up to "first TID" from the next tuple, and uses
that to write data into index if needed because of memory limit.
We don't want to do that too often, so it only happens if we hit the
memory limit and there's at least a certain number (1024) of TIDs.
7) v20240502-0007-Detect-wrap-around-in-parallel-callback.patch
There's one more efficiency problem - the parallel scans are required to
be synchronized, i.e. the scan may start half-way through the table, and
then wrap around. Which however means the TID list will have a very wide
range of TID values, essentially the min and max of for the key.
Without 0006 this would cause frequent failures of the index build, with
the error I already mentioned:
ERROR: could not split GIN page; all old items didn't fit
tracking the "safe" TID horizon addresses that. But there's still an
issue with efficiency - having such a wide TID list forces the mergesort
to actually walk the lists, because this wide list overlaps with every
other list produced by the worker. And that's much more expensive than
just simply concatenating them, which is what happens without the wrap
around (because in that case the worker produces non-overlapping lists).
One way to fix this would be to allow parallel scans to not be sync
scans, but that seems fairly tricky and I'm not sure if that can be
done. The BRIN parallel build had a similar issue, and it was just
simpler to deal with this in the build code.
So 0007 does something similar - it tracks if the TID value goes
backward in the callback, and if it does it dumps the state into the
tuplesort before processing the first tuple from the beginning of the
table. Which means we end up with two separate "narrow" TID list, not
one very wide one.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[image/png] gin-parallel-absolute.png (17.2K, ../../[email protected]/2-gin-parallel-absolute.png)
download | view image
[image/png] gin-parallel-relative.png (23.7K, ../../[email protected]/3-gin-parallel-relative.png)
download | view image
[text/x-patch] v20240502-0001-Allow-parallel-create-for-GIN-indexes.patch (56.2K, ../../[email protected]/4-v20240502-0001-Allow-parallel-create-for-GIN-indexes.patch)
download | inline diff:
From 35cf84ee568df2cb7eb4027dcf84fefd02e45509 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:26 +0200
Subject: [PATCH v20240502 1/8] Allow parallel create for GIN indexes
Add support for parallel create of GIN indexes, using an approach and
code very similar to the one used by BRIN indexes.
Each worker reads a subset of the table (from a parallel scan), and
accumulated index entries in memory. But instead of writing the results
into the index (after hitting the memory limit), the data are written
to a shared tuplesort (and sorted by index key).
The leader then reads data from the tuplesort, and combines them into
entries that get inserted into the index.
---
src/backend/access/gin/ginbulk.c | 7 +
src/backend/access/gin/gininsert.c | 1335 +++++++++++++++++++-
src/backend/access/gin/ginutil.c | 2 +-
src/backend/access/transam/parallel.c | 4 +
src/backend/utils/sort/tuplesortvariants.c | 154 +++
src/include/access/gin.h | 4 +
src/include/access/gin_tuple.h | 28 +
src/include/utils/tuplesort.h | 6 +
src/tools/pgindent/typedefs.list | 4 +
9 files changed, 1528 insertions(+), 16 deletions(-)
create mode 100644 src/include/access/gin_tuple.h
diff --git a/src/backend/access/gin/ginbulk.c b/src/backend/access/gin/ginbulk.c
index a522801c2f7..12eeff04a6c 100644
--- a/src/backend/access/gin/ginbulk.c
+++ b/src/backend/access/gin/ginbulk.c
@@ -153,6 +153,13 @@ ginInsertBAEntry(BuildAccumulator *accum,
GinEntryAccumulator *ea;
bool isNew;
+ /*
+ * FIXME prevents writes of uninitialized bytes reported by valgrind in
+ * writetup (likely that build_gin_tuple copies some fields that are only
+ * initialized for a certain category, or something similar)
+ */
+ memset(&eatmp, 0, sizeof(GinEntryAccumulator));
+
/*
* For the moment, fill only the fields of eatmp that will be looked at by
* cmpEntryAccumulator or ginCombineData.
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 71f38be90c3..4d6d152403e 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -15,14 +15,124 @@
#include "postgres.h"
#include "access/gin_private.h"
+#include "access/gin_tuple.h"
+#include "access/table.h"
#include "access/tableam.h"
#include "access/xloginsert.h"
+#include "catalog/index.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "nodes/execnodes.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "tcop/tcopprot.h" /* pgrminclude ignore */
+#include "utils/datum.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/builtins.h"
+
+
+/* Magic numbers for parallel state sharing */
+#define PARALLEL_KEY_GIN_SHARED UINT64CONST(0xB000000000000001)
+#define PARALLEL_KEY_TUPLESORT UINT64CONST(0xB000000000000002)
+#define PARALLEL_KEY_QUERY_TEXT UINT64CONST(0xB000000000000003)
+#define PARALLEL_KEY_WAL_USAGE UINT64CONST(0xB000000000000004)
+#define PARALLEL_KEY_BUFFER_USAGE UINT64CONST(0xB000000000000005)
+
+/*
+ * Status for index builds performed in parallel. This is allocated in a
+ * dynamic shared memory segment.
+ */
+typedef struct GinShared
+{
+ /*
+ * These fields are not modified during the build. They primarily exist
+ * for the benefit of worker processes that need to create state
+ * corresponding to that used by the leader.
+ */
+ Oid heaprelid;
+ Oid indexrelid;
+ bool isconcurrent;
+ int scantuplesortstates;
+
+ /*
+ * workersdonecv is used to monitor the progress of workers. All parallel
+ * participants must indicate that they are done before leader can use
+ * results built by the workers (and before leader can write the data into
+ * the index).
+ */
+ ConditionVariable workersdonecv;
+
+ /*
+ * mutex protects all fields before heapdesc.
+ *
+ * These fields contain status information of interest to GIN index builds
+ * that must work just the same when an index is built in parallel.
+ */
+ slock_t mutex;
+
+ /*
+ * Mutable state that is maintained by workers, and reported back to
+ * leader at end of the scans.
+ *
+ * nparticipantsdone is number of worker processes finished.
+ *
+ * reltuples is the total number of input heap tuples.
+ *
+ * indtuples is the total number of tuples that made it into the index.
+ */
+ int nparticipantsdone;
+ double reltuples;
+ double indtuples;
+
+ /*
+ * ParallelTableScanDescData data follows. Can't directly embed here, as
+ * implementations of the parallel table scan desc interface might need
+ * stronger alignment.
+ */
+} GinShared;
+
+/*
+ * Return pointer to a GinShared's parallel table scan.
+ *
+ * c.f. shm_toc_allocate as to why BUFFERALIGN is used, rather than just
+ * MAXALIGN.
+ */
+#define ParallelTableScanFromGinShared(shared) \
+ (ParallelTableScanDesc) ((char *) (shared) + BUFFERALIGN(sizeof(GinShared)))
+
+/*
+ * Status for leader in parallel index build.
+ */
+typedef struct GinLeader
+{
+ /* parallel context itself */
+ ParallelContext *pcxt;
+
+ /*
+ * nparticipanttuplesorts is the exact number of worker processes
+ * successfully launched, plus one leader process if it participates as a
+ * worker (only DISABLE_LEADER_PARTICIPATION builds avoid leader
+ * participating as a worker).
+ */
+ int nparticipanttuplesorts;
+
+ /*
+ * Leader process convenience pointers to shared state (leader avoids TOC
+ * lookups).
+ *
+ * GinShared is the shared state for entire build. sharedsort is the
+ * shared, tuplesort-managed state passed to each process tuplesort.
+ * snapshot is the snapshot used by the scan iff an MVCC snapshot is
+ * required.
+ */
+ GinShared *ginshared;
+ Sharedsort *sharedsort;
+ Snapshot snapshot;
+ WalUsage *walusage;
+ BufferUsage *bufferusage;
+} GinLeader;
typedef struct
{
@@ -32,9 +142,49 @@ typedef struct
MemoryContext tmpCtx;
MemoryContext funcCtx;
BuildAccumulator accum;
+
+ /* FIXME likely duplicate with indtuples */
+ double bs_numtuples;
+ double bs_reltuples;
+
+ /*
+ * bs_leader is only present when a parallel index build is performed, and
+ * only in the leader process. (Actually, only the leader process has a
+ * GinBuildState.)
+ */
+ GinLeader *bs_leader;
+ int bs_worker_id;
+
+ /*
+ * The sortstate is used by workers (including the leader). It has to be
+ * part of the build state, because that's the only thing passed to the
+ * build callback etc.
+ */
+ Tuplesortstate *bs_sortstate;
} GinBuildState;
+/* parallel index builds */
+static void _gin_begin_parallel(GinBuildState *buildstate, Relation heap, Relation index,
+ bool isconcurrent, int request);
+static void _gin_end_parallel(GinLeader *ginleader, GinBuildState *state);
+static Size _gin_parallel_estimate_shared(Relation heap, Snapshot snapshot);
+static double _gin_parallel_heapscan(GinBuildState *buildstate);
+static double _gin_parallel_merge(GinBuildState *buildstate);
+static void _gin_leader_participate_as_worker(GinBuildState *buildstate,
+ Relation heap, Relation index);
+static void _gin_parallel_scan_and_build(GinBuildState *buildstate,
+ GinShared *ginshared,
+ Sharedsort *sharedsort,
+ Relation heap, Relation index,
+ int sortmem, bool progress);
+
+static Datum _gin_parse_tuple(GinTuple *a, ItemPointerData **items);
+static GinTuple *_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+ Datum key, int16 typlen, bool typbyval,
+ ItemPointerData *items, uint32 nitems,
+ Size *len);
+
/*
* Adds array of item pointers to tuple's posting list, or
* creates posting tree and tuple pointing to tree in case
@@ -313,12 +463,95 @@ ginBuildCallback(Relation index, ItemPointer tid, Datum *values,
MemoryContextSwitchTo(oldCtx);
}
+static void
+ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
+ bool *isnull, bool tupleIsAlive, void *state)
+{
+ GinBuildState *buildstate = (GinBuildState *) state;
+ MemoryContext oldCtx;
+ int i;
+
+ oldCtx = MemoryContextSwitchTo(buildstate->tmpCtx);
+
+ for (i = 0; i < buildstate->ginstate.origTupdesc->natts; i++)
+ ginHeapTupleBulkInsert(buildstate, (OffsetNumber) (i + 1),
+ values[i], isnull[i], tid);
+
+ /*
+ * XXX idea - Instead of writing the entries directly into the shared
+ * tuplesort, write it into a local one, do the sort in the worker, and
+ * combine the results. For large tables with many different keys that's
+ * going to work better than the current approach where we don't get many
+ * matches in work_mem (maybe this should use 32MB, which is what we use
+ * when planning, but even that may not be great). Which means we are
+ * likely to have many entries with a single TID, forcing the leader to do
+ * a qsort() when merging the data, often amounting to ~50% of the serial
+ * part. By doing the qsort() in a worker, leader then can do a mergesort
+ * (likely cheaper). Also, it means the amount of data worker->leader is
+ * going to be lower thanks to deduplication.
+ *
+ * Disadvantage: It needs more disk space, possibly up to 2x, because each
+ * worker creates a tuplestore, then "transforms it" into the shared
+ * tuplestore (hopefully less data, but not guaranteed).
+ *
+ * It's however possible to partition the data into multiple tuplesorts
+ * per worker (by hashing). We don't need perfect sorting, and we can even
+ * live with "equal" keys having multiple hashes (if there are multiple
+ * binary representations of the value).
+ */
+
+ /*
+ * If we've maxed out our available memory, dump everything to the
+ * tuplesort
+ *
+ * XXX probably should use 32MB, not work_mem, as used during planning?
+ */
+ if (buildstate->accum.allocatedMemory >= (Size) work_mem * 1024L)
+ {
+ ItemPointerData *list;
+ Datum key;
+ GinNullCategory category;
+ uint32 nlist;
+ OffsetNumber attnum;
+ TupleDesc tdesc = RelationGetDescr(index);
+
+ ginBeginBAScan(&buildstate->accum);
+ while ((list = ginGetBAEntry(&buildstate->accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* information about the key */
+ Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
+
+ /* GIN tuple and tuple length */
+ GinTuple *tup;
+ Size tuplen;
+
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+
+ tup = _gin_build_tuple(attnum, category,
+ key, attr->attlen, attr->attbyval,
+ list, nlist, &tuplen);
+
+ tuplesort_putgintuple(buildstate->bs_sortstate, tup, tuplen);
+
+ pfree(tup);
+ }
+
+ MemoryContextReset(buildstate->tmpCtx);
+ ginInitBA(&buildstate->accum);
+ }
+
+ MemoryContextSwitchTo(oldCtx);
+}
+
IndexBuildResult *
ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
{
IndexBuildResult *result;
double reltuples;
GinBuildState buildstate;
+ GinBuildState *state = &buildstate;
Buffer RootBuffer,
MetaBuffer;
ItemPointerData *list;
@@ -336,6 +569,14 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.indtuples = 0;
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
+ /*
+ * XXX Make sure to initialize a bunch of fields, not to trip valgrind.
+ * Maybe there should be an "init" function for build state?
+ */
+ buildstate.bs_numtuples = 0;
+ buildstate.bs_reltuples = 0;
+ buildstate.bs_leader = NULL;
+
/* initialize the meta page */
MetaBuffer = GinNewBuffer(index);
@@ -376,25 +617,91 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
ginInitBA(&buildstate.accum);
/*
- * Do the heap scan. We disallow sync scan here because dataPlaceToPage
- * prefers to receive tuples in TID order.
+ * Attempt to launch parallel worker scan when required
+ *
+ * XXX plan_create_index_workers makes the number of workers dependent on
+ * maintenance_work_mem, requiring 32MB for each worker. That makes sense
+ * for btree, but not for GIN, which can do with much less memory. So
+ * maybe make that somehow less strict, optionally?
+ */
+ if (indexInfo->ii_ParallelWorkers > 0)
+ _gin_begin_parallel(state, heap, index, indexInfo->ii_Concurrent,
+ indexInfo->ii_ParallelWorkers);
+
+
+ /*
+ * If parallel build requested and at least one worker process was
+ * successfully launched, set up coordination state, wait for workers to
+ * complete. Then read all tuples from the shared tuplesort and insert
+ * them into the index.
+ *
+ * In serial mode, simply scan the table and build the index one index
+ * tuple at a time.
*/
- reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
- ginBuildCallback, (void *) &buildstate,
- NULL);
+ if (state->bs_leader)
+ {
+ SortCoordinate coordinate;
+
+ coordinate = (SortCoordinate) palloc0(sizeof(SortCoordinateData));
+ coordinate->isWorker = false;
+ coordinate->nParticipants =
+ state->bs_leader->nparticipanttuplesorts;
+ coordinate->sharedsort = state->bs_leader->sharedsort;
+
+ /*
+ * Begin leader tuplesort.
+ *
+ * In cases where parallelism is involved, the leader receives the
+ * same share of maintenance_work_mem as a serial sort (it is
+ * generally treated in the same way as a serial sort once we return).
+ * Parallel worker Tuplesortstates will have received only a fraction
+ * of maintenance_work_mem, though.
+ *
+ * We rely on the lifetime of the Leader Tuplesortstate almost not
+ * overlapping with any worker Tuplesortstate's lifetime. There may
+ * be some small overlap, but that's okay because we rely on leader
+ * Tuplesortstate only allocating a small, fixed amount of memory
+ * here. When its tuplesort_performsort() is called (by our caller),
+ * and significant amounts of memory are likely to be used, all
+ * workers must have already freed almost all memory held by their
+ * Tuplesortstates (they are about to go away completely, too). The
+ * overall effect is that maintenance_work_mem always represents an
+ * absolute high watermark on the amount of memory used by a CREATE
+ * INDEX operation, regardless of the use of parallelism or any other
+ * factor.
+ */
+ state->bs_sortstate =
+ tuplesort_begin_index_gin(maintenance_work_mem, coordinate,
+ TUPLESORT_NONE);
- /* dump remaining entries to the index */
- oldCtx = MemoryContextSwitchTo(buildstate.tmpCtx);
- ginBeginBAScan(&buildstate.accum);
- while ((list = ginGetBAEntry(&buildstate.accum,
- &attnum, &key, &category, &nlist)) != NULL)
+ /* scan the relation and merge per-worker results */
+ reltuples = _gin_parallel_merge(state);
+
+ _gin_end_parallel(state->bs_leader, state);
+ }
+ else /* no parallel index build */
{
- /* there could be many entries, so be willing to abort here */
- CHECK_FOR_INTERRUPTS();
- ginEntryInsert(&buildstate.ginstate, attnum, key, category,
- list, nlist, &buildstate.buildStats);
+ /*
+ * Do the heap scan. We disallow sync scan here because
+ * dataPlaceToPage prefers to receive tuples in TID order.
+ */
+ reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
+ ginBuildCallback, (void *) &buildstate,
+ NULL);
+
+ /* dump remaining entries to the index */
+ oldCtx = MemoryContextSwitchTo(buildstate.tmpCtx);
+ ginBeginBAScan(&buildstate.accum);
+ while ((list = ginGetBAEntry(&buildstate.accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+ ginEntryInsert(&buildstate.ginstate, attnum, key, category,
+ list, nlist, &buildstate.buildStats);
+ }
+ MemoryContextSwitchTo(oldCtx);
}
- MemoryContextSwitchTo(oldCtx);
MemoryContextDelete(buildstate.funcCtx);
MemoryContextDelete(buildstate.tmpCtx);
@@ -534,3 +841,1001 @@ gininsert(Relation index, Datum *values, bool *isnull,
return false;
}
+
+/*
+ * Create parallel context, and launch workers for leader.
+ *
+ * buildstate argument should be initialized (with the exception of the
+ * tuplesort states, which may later be created based on shared
+ * state initially set up here).
+ *
+ * isconcurrent indicates if operation is CREATE INDEX CONCURRENTLY.
+ *
+ * request is the target number of parallel worker processes to launch.
+ *
+ * Sets buildstate's GinLeader, which caller must use to shut down parallel
+ * mode by passing it to _gin_end_parallel() at the very end of its index
+ * build. If not even a single worker process can be launched, this is
+ * never set, and caller should proceed with a serial index build.
+ */
+static void
+_gin_begin_parallel(GinBuildState *buildstate, Relation heap, Relation index,
+ bool isconcurrent, int request)
+{
+ ParallelContext *pcxt;
+ int scantuplesortstates;
+ Snapshot snapshot;
+ Size estginshared;
+ Size estsort;
+ GinShared *ginshared;
+ Sharedsort *sharedsort;
+ GinLeader *ginleader = (GinLeader *) palloc0(sizeof(GinLeader));
+ WalUsage *walusage;
+ BufferUsage *bufferusage;
+ bool leaderparticipates = true;
+ int querylen;
+
+#ifdef DISABLE_LEADER_PARTICIPATION
+ leaderparticipates = false;
+#endif
+
+ /*
+ * Enter parallel mode, and create context for parallel build of gin index
+ */
+ EnterParallelMode();
+ Assert(request > 0);
+ pcxt = CreateParallelContext("postgres", "_gin_parallel_build_main",
+ request);
+
+ scantuplesortstates = leaderparticipates ? request + 1 : request;
+
+ /*
+ * Prepare for scan of the base relation. In a normal index build, we use
+ * SnapshotAny because we must retrieve all tuples and do our own time
+ * qual checks (because we have to index RECENTLY_DEAD tuples). In a
+ * concurrent build, we take a regular MVCC snapshot and index whatever's
+ * live according to that.
+ */
+ if (!isconcurrent)
+ snapshot = SnapshotAny;
+ else
+ snapshot = RegisterSnapshot(GetTransactionSnapshot());
+
+ /*
+ * Estimate size for our own PARALLEL_KEY_GIN_SHARED workspace.
+ */
+ estginshared = _gin_parallel_estimate_shared(heap, snapshot);
+ shm_toc_estimate_chunk(&pcxt->estimator, estginshared);
+ estsort = tuplesort_estimate_shared(scantuplesortstates);
+ shm_toc_estimate_chunk(&pcxt->estimator, estsort);
+
+ shm_toc_estimate_keys(&pcxt->estimator, 2);
+
+ /*
+ * Estimate space for WalUsage and BufferUsage -- PARALLEL_KEY_WAL_USAGE
+ * and PARALLEL_KEY_BUFFER_USAGE.
+ *
+ * If there are no extensions loaded that care, we could skip this. We
+ * have no way of knowing whether anyone's looking at pgWalUsage or
+ * pgBufferUsage, so do it unconditionally.
+ */
+ shm_toc_estimate_chunk(&pcxt->estimator,
+ mul_size(sizeof(WalUsage), pcxt->nworkers));
+ shm_toc_estimate_keys(&pcxt->estimator, 1);
+ shm_toc_estimate_chunk(&pcxt->estimator,
+ mul_size(sizeof(BufferUsage), pcxt->nworkers));
+ shm_toc_estimate_keys(&pcxt->estimator, 1);
+
+ /* Finally, estimate PARALLEL_KEY_QUERY_TEXT space */
+ if (debug_query_string)
+ {
+ querylen = strlen(debug_query_string);
+ shm_toc_estimate_chunk(&pcxt->estimator, querylen + 1);
+ shm_toc_estimate_keys(&pcxt->estimator, 1);
+ }
+ else
+ querylen = 0; /* keep compiler quiet */
+
+ /* Everyone's had a chance to ask for space, so now create the DSM */
+ InitializeParallelDSM(pcxt);
+
+ /* If no DSM segment was available, back out (do serial build) */
+ if (pcxt->seg == NULL)
+ {
+ if (IsMVCCSnapshot(snapshot))
+ UnregisterSnapshot(snapshot);
+ DestroyParallelContext(pcxt);
+ ExitParallelMode();
+ return;
+ }
+
+ /* Store shared build state, for which we reserved space */
+ ginshared = (GinShared *) shm_toc_allocate(pcxt->toc, estginshared);
+ /* Initialize immutable state */
+ ginshared->heaprelid = RelationGetRelid(heap);
+ ginshared->indexrelid = RelationGetRelid(index);
+ ginshared->isconcurrent = isconcurrent;
+ ginshared->scantuplesortstates = scantuplesortstates;
+
+ ConditionVariableInit(&ginshared->workersdonecv);
+ SpinLockInit(&ginshared->mutex);
+
+ /* Initialize mutable state */
+ ginshared->nparticipantsdone = 0;
+ ginshared->reltuples = 0.0;
+ ginshared->indtuples = 0.0;
+
+ table_parallelscan_initialize(heap,
+ ParallelTableScanFromGinShared(ginshared),
+ snapshot);
+
+ /*
+ * Store shared tuplesort-private state, for which we reserved space.
+ * Then, initialize opaque state using tuplesort routine.
+ */
+ sharedsort = (Sharedsort *) shm_toc_allocate(pcxt->toc, estsort);
+ tuplesort_initialize_shared(sharedsort, scantuplesortstates,
+ pcxt->seg);
+
+ /*
+ * Store shared tuplesort-private state, for which we reserved space.
+ * Then, initialize opaque state using tuplesort routine.
+ */
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_GIN_SHARED, ginshared);
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_TUPLESORT, sharedsort);
+
+ /* Store query string for workers */
+ if (debug_query_string)
+ {
+ char *sharedquery;
+
+ sharedquery = (char *) shm_toc_allocate(pcxt->toc, querylen + 1);
+ memcpy(sharedquery, debug_query_string, querylen + 1);
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_QUERY_TEXT, sharedquery);
+ }
+
+ /*
+ * Allocate space for each worker's WalUsage and BufferUsage; no need to
+ * initialize.
+ */
+ walusage = shm_toc_allocate(pcxt->toc,
+ mul_size(sizeof(WalUsage), pcxt->nworkers));
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_WAL_USAGE, walusage);
+ bufferusage = shm_toc_allocate(pcxt->toc,
+ mul_size(sizeof(BufferUsage), pcxt->nworkers));
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_BUFFER_USAGE, bufferusage);
+
+ /* Launch workers, saving status for leader/caller */
+ LaunchParallelWorkers(pcxt);
+ ginleader->pcxt = pcxt;
+ ginleader->nparticipanttuplesorts = pcxt->nworkers_launched;
+ if (leaderparticipates)
+ ginleader->nparticipanttuplesorts++;
+ ginleader->ginshared = ginshared;
+ ginleader->sharedsort = sharedsort;
+ ginleader->snapshot = snapshot;
+ ginleader->walusage = walusage;
+ ginleader->bufferusage = bufferusage;
+
+ /* If no workers were successfully launched, back out (do serial build) */
+ if (pcxt->nworkers_launched == 0)
+ {
+ _gin_end_parallel(ginleader, NULL);
+ return;
+ }
+
+ /* Save leader state now that it's clear build will be parallel */
+ buildstate->bs_leader = ginleader;
+
+ /* Join heap scan ourselves */
+ if (leaderparticipates)
+ _gin_leader_participate_as_worker(buildstate, heap, index);
+
+ /*
+ * Caller needs to wait for all launched workers when we return. Make
+ * sure that the failure-to-start case will not hang forever.
+ */
+ WaitForParallelWorkersToAttach(pcxt);
+}
+
+/*
+ * Shut down workers, destroy parallel context, and end parallel mode.
+ */
+static void
+_gin_end_parallel(GinLeader *ginleader, GinBuildState *state)
+{
+ int i;
+
+ /* Shutdown worker processes */
+ WaitForParallelWorkersToFinish(ginleader->pcxt);
+
+ /*
+ * Next, accumulate WAL usage. (This must wait for the workers to finish,
+ * or we might get incomplete data.)
+ */
+ for (i = 0; i < ginleader->pcxt->nworkers_launched; i++)
+ InstrAccumParallelQuery(&ginleader->bufferusage[i], &ginleader->walusage[i]);
+
+ /* Free last reference to MVCC snapshot, if one was used */
+ if (IsMVCCSnapshot(ginleader->snapshot))
+ UnregisterSnapshot(ginleader->snapshot);
+ DestroyParallelContext(ginleader->pcxt);
+ ExitParallelMode();
+}
+
+/*
+ * Within leader, wait for end of heap scan.
+ *
+ * When called, parallel heap scan started by _gin_begin_parallel() will
+ * already be underway within worker processes (when leader participates
+ * as a worker, we should end up here just as workers are finishing).
+ *
+ * Returns the total number of heap tuples scanned.
+ */
+static double
+_gin_parallel_heapscan(GinBuildState *state)
+{
+ GinShared *ginshared = state->bs_leader->ginshared;
+ int nparticipanttuplesorts;
+
+ nparticipanttuplesorts = state->bs_leader->nparticipanttuplesorts;
+ for (;;)
+ {
+ SpinLockAcquire(&ginshared->mutex);
+ if (ginshared->nparticipantsdone == nparticipanttuplesorts)
+ {
+ /* copy the data into leader state */
+ state->bs_reltuples = ginshared->reltuples;
+ state->bs_numtuples = ginshared->indtuples;
+
+ SpinLockRelease(&ginshared->mutex);
+ break;
+ }
+ SpinLockRelease(&ginshared->mutex);
+
+ ConditionVariableSleep(&ginshared->workersdonecv,
+ WAIT_EVENT_PARALLEL_CREATE_INDEX_SCAN);
+ }
+
+ ConditionVariableCancelSleep();
+
+ return state->bs_reltuples;
+}
+
+static int
+tid_cmp(const void *a, const void *b)
+{
+ return ItemPointerCompare((ItemPointer) a, (ItemPointer) b);
+}
+
+/*
+ * State used to combine accumulate TIDs from multiple GinTuples for the same
+ * key value.
+ *
+ * XXX Similar purpose to BuildAccumulator, but much simpler.
+ */
+typedef struct GinBuffer
+{
+ OffsetNumber attnum;
+ GinNullCategory category;
+ Datum key; /* 0 if no key (and keylen == 0) */
+ Size keylen; /* number of bytes (not typlen) */
+
+ /* type info */
+ int16 typlen;
+ bool typbyval;
+
+ /* array of TID values */
+ int nitems;
+ int maxitems;
+ ItemPointerData *items;
+} GinBuffer;
+
+/* XXX should do more checks */
+static void
+AssertCheckGinBuffer(GinBuffer *buffer)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(buffer->nitems <= buffer->maxitems);
+#endif
+}
+
+static void
+AssertCheckItemPointers(ItemPointerData *items, int nitems, bool sorted)
+{
+#ifdef USE_ASSERT_CHECKING
+ for (int i = 0; i < nitems; i++)
+ {
+ Assert(ItemPointerIsValid(&items[i]));
+
+ if ((i == 0) || !sorted)
+ continue;
+
+ Assert(ItemPointerCompare(&items[i - 1], &items[i]) < 0);
+ }
+#endif
+}
+
+static GinBuffer *
+GinBufferInit(void)
+{
+ return palloc0(sizeof(GinBuffer));
+}
+
+static bool
+GinBufferIsEmpty(GinBuffer *buffer)
+{
+ return (buffer->nitems == 0);
+}
+
+/*
+ * Compare if the tuple matches the already accumulated data. Compare
+ * scalar fields first, before the actual key.
+ *
+ * XXX The key is compared using memcmp, which means that if a key has
+ * multiple binary representations, we may end up treating them as
+ * different here. But that's OK, the index will merge them anyway.
+ */
+static bool
+GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
+{
+ AssertCheckGinBuffer(buffer);
+
+ if (tup->attrnum != buffer->attnum)
+ return false;
+
+ /* same attribute should have the same type info */
+ Assert(tup->typbyval == buffer->typbyval);
+ Assert(tup->typlen == buffer->typlen);
+
+ if (tup->category != buffer->category)
+ return false;
+
+ if (tup->keylen != buffer->keylen)
+ return false;
+
+ /*
+ * For NULL/empty keys, this means equality, for normal keys we need to
+ * compare the actual key value.
+ */
+ if (buffer->category != GIN_CAT_NORM_KEY)
+ return true;
+
+ /*
+ * Compare the key value, depending on the type information.
+ *
+ * XXX Not sure this works correctly for byval types that don't need the
+ * whole Datum. What if there is garbage in the padding bytes?
+ */
+ if (buffer->typbyval)
+ return (buffer->key == *(Datum *) tup->data);
+
+ /* byref values simply uses memcmp for comparison */
+ return (memcmp(tup->data, DatumGetPointer(buffer->key), buffer->keylen) == 0);
+}
+
+static void
+GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
+{
+ ItemPointerData *items;
+ Datum key;
+
+ AssertCheckGinBuffer(buffer);
+
+ key = _gin_parse_tuple(tup, &items);
+
+ /* if the buffer is empty, set the fields (and copy the key) */
+ if (GinBufferIsEmpty(buffer))
+ {
+ buffer->category = tup->category;
+ buffer->keylen = tup->keylen;
+ buffer->attnum = tup->attrnum;
+
+ buffer->typlen = tup->typlen;
+ buffer->typbyval = tup->typbyval;
+
+ if (tup->category == GIN_CAT_NORM_KEY)
+ buffer->key = datumCopy(key, buffer->typbyval, buffer->typlen);
+ else
+ buffer->key = (Datum) 0;
+ }
+
+ /* enlarge the TID buffer, if needed */
+ if (buffer->nitems + tup->nitems > buffer->maxitems)
+ {
+ /* 64 seems like a good init value */
+ buffer->maxitems = Max(buffer->maxitems, 64);
+
+ while (buffer->nitems + tup->nitems > buffer->maxitems)
+ buffer->maxitems *= 2;
+
+ if (buffer->items == NULL)
+ buffer->items = palloc(buffer->maxitems * sizeof(ItemPointerData));
+ else
+ buffer->items = repalloc(buffer->items,
+ buffer->maxitems * sizeof(ItemPointerData));
+ }
+
+ /* now we should be guaranteed to have enough space for all the TIDs */
+ Assert(buffer->nitems + tup->nitems <= buffer->maxitems);
+
+ /* copy the new TIDs into the buffer */
+ memcpy(&buffer->items[buffer->nitems], items, sizeof(ItemPointerData) * tup->nitems);
+ buffer->nitems += tup->nitems;
+
+ AssertCheckItemPointers(buffer->items, buffer->nitems, false);
+}
+
+static void
+GinBufferSortItems(GinBuffer *buffer)
+{
+ /* we should not have a buffer with no TIDs to sort */
+ Assert(buffer->items != NULL);
+ Assert(buffer->nitems > 0);
+
+ pg_qsort(buffer->items, buffer->nitems, sizeof(ItemPointerData), tid_cmp);
+
+ AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+}
+
+/* XXX probably would be better to have a memory context for the buffer */
+static void
+GinBufferReset(GinBuffer *buffer)
+{
+ Assert(!GinBufferIsEmpty(buffer));
+
+ /* release byref values, do nothing for by-val ones */
+ if ((buffer->category == GIN_CAT_NORM_KEY) && !buffer->typbyval)
+ pfree(DatumGetPointer(buffer->key));
+
+ /* XXX not really needed, but easier to trigger NULL deref etc. */
+ buffer->key = (Datum) 0;
+
+ buffer->attnum = 0;
+ buffer->category = 0;
+ buffer->keylen = 0;
+ buffer->nitems = 0;
+
+ buffer->typlen = 0;
+ buffer->typbyval = 0;
+
+ /* XXX should do something with extremely large array of items? */
+}
+
+/*
+ * XXX Maybe check size of the TID arrays, and return false if it's too
+ * large (more thant maintenance_work_mem or something?).
+ */
+static bool
+GinBufferCanAddKey(GinBuffer *buffer, GinTuple *tup)
+{
+ /* empty buffer can accept data for any key */
+ if (GinBufferIsEmpty(buffer))
+ return true;
+
+ /* otherwise just data for the same key */
+ return GinBufferKeyEquals(buffer, tup);
+}
+
+/*
+ * Within leader, wait for end of heap scan and merge per-worker results.
+ *
+ * After waiting for all workers to finish, merge the per-worker results into
+ * the complete index. The results from each worker are sorted by block number
+ * (start of the page range). While combinig the per-worker results we merge
+ * summaries for the same page range, and also fill-in empty summaries for
+ * ranges without any tuples.
+ *
+ * Returns the total number of heap tuples scanned.
+ *
+ * FIXME probably should have local memory contexts similar to what
+ * _brin_parallel_merge does.
+ */
+static double
+_gin_parallel_merge(GinBuildState *state)
+{
+ GinTuple *tup;
+ Size tuplen;
+ double reltuples = 0;
+ GinBuffer *buffer;
+
+ /* wait for workers to scan table and produce partial results */
+ reltuples = _gin_parallel_heapscan(state);
+
+ /* do the actual sort in the leader */
+ tuplesort_performsort(state->bs_sortstate);
+
+ /* initialize buffer to combine entries for the same key */
+ buffer = GinBufferInit();
+
+ /*
+ * Read the GIN tuples from the shared tuplesort, sorted by category and
+ * key. That probably gives us order matching how data is organized in the
+ * index.
+ *
+ * XXX Maybe we should sort by key first, then by category?
+ *
+ * We don't insert the GIN tuples right away, but instead accumulate as
+ * many TIDs for the same key as possible, and then insert that at once.
+ * This way we don't need to decompress/recompress the posting lists, etc.
+ */
+ while ((tup = tuplesort_getgintuple(state->bs_sortstate, &tuplen, true)) != NULL)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * If the buffer can accept the new GIN tuple, just store it there and
+ * we're done. If it's a different key (or maybe too much data) flush
+ * the current contents into the index first.
+ */
+ if (!GinBufferCanAddKey(buffer, tup))
+ {
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ GinBufferSortItems(buffer);
+
+ ginEntryInsert(&state->ginstate,
+ buffer->attnum, buffer->key, buffer->category,
+ buffer->items, buffer->nitems, &state->buildStats);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ /* now remember the new key */
+ GinBufferStoreTuple(buffer, tup);
+ }
+
+ /* flush data remaining in the buffer (for the last key) */
+ if (!GinBufferIsEmpty(buffer))
+ {
+ GinBufferSortItems(buffer);
+
+ ginEntryInsert(&state->ginstate,
+ buffer->attnum, buffer->key, buffer->category,
+ buffer->items, buffer->nitems, &state->buildStats);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ tuplesort_end(state->bs_sortstate);
+
+ return reltuples;
+}
+
+/*
+ * Returns size of shared memory required to store state for a parallel
+ * gin index build based on the snapshot its parallel scan will use.
+ */
+static Size
+_gin_parallel_estimate_shared(Relation heap, Snapshot snapshot)
+{
+ /* c.f. shm_toc_allocate as to why BUFFERALIGN is used */
+ return add_size(BUFFERALIGN(sizeof(GinShared)),
+ table_parallelscan_estimate(heap, snapshot));
+}
+
+/*
+ * Within leader, participate as a parallel worker.
+ */
+static void
+_gin_leader_participate_as_worker(GinBuildState *buildstate, Relation heap, Relation index)
+{
+ GinLeader *ginleader = buildstate->bs_leader;
+ int sortmem;
+
+ /*
+ * Might as well use reliable figure when doling out maintenance_work_mem
+ * (when requested number of workers were not launched, this will be
+ * somewhat higher than it is for other workers).
+ */
+ sortmem = maintenance_work_mem / ginleader->nparticipanttuplesorts;
+
+ /* Perform work common to all participants */
+ _gin_parallel_scan_and_build(buildstate, ginleader->ginshared,
+ ginleader->sharedsort, heap, index, sortmem, true);
+}
+
+/*
+ * Perform a worker's portion of a parallel sort.
+ *
+ * This generates a tuplesort for the worker portion of the table.
+ *
+ * sortmem is the amount of working memory to use within each worker,
+ * expressed in KBs.
+ *
+ * When this returns, workers are done, and need only release resources.
+ */
+static void
+_gin_parallel_scan_and_build(GinBuildState *state,
+ GinShared *ginshared, Sharedsort *sharedsort,
+ Relation heap, Relation index,
+ int sortmem, bool progress)
+{
+ SortCoordinate coordinate;
+ TableScanDesc scan;
+ double reltuples;
+ IndexInfo *indexInfo;
+
+ /* Initialize local tuplesort coordination state */
+ coordinate = palloc0(sizeof(SortCoordinateData));
+ coordinate->isWorker = true;
+ coordinate->nParticipants = -1;
+ coordinate->sharedsort = sharedsort;
+
+ /* Begin "partial" tuplesort */
+ state->bs_sortstate = tuplesort_begin_index_gin(sortmem, coordinate,
+ TUPLESORT_NONE);
+
+ /* Join parallel scan */
+ indexInfo = BuildIndexInfo(index);
+ indexInfo->ii_Concurrent = ginshared->isconcurrent;
+
+ scan = table_beginscan_parallel(heap,
+ ParallelTableScanFromGinShared(ginshared));
+
+ reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
+ ginBuildCallbackParallel, state, scan);
+
+ /* insert the last item */
+ /* write remaining accumulated entries */
+ {
+ ItemPointerData *list;
+ Datum key;
+ GinNullCategory category;
+ uint32 nlist;
+ OffsetNumber attnum;
+ TupleDesc tdesc = RelationGetDescr(index);
+
+ ginBeginBAScan(&state->accum);
+ while ((list = ginGetBAEntry(&state->accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* information about the key */
+ Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
+
+ GinTuple *tup;
+ Size len;
+
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+
+ tup = _gin_build_tuple(attnum, category,
+ key, attr->attlen, attr->attbyval,
+ list, nlist, &len);
+
+ tuplesort_putgintuple(state->bs_sortstate, tup, len);
+
+ pfree(tup);
+ }
+
+ MemoryContextReset(state->tmpCtx);
+ ginInitBA(&state->accum);
+ }
+
+ /* sort the GIN tuples built by this worker */
+ tuplesort_performsort(state->bs_sortstate);
+
+ state->bs_reltuples += reltuples;
+
+ /*
+ * Done. Record ambuild statistics.
+ */
+ SpinLockAcquire(&ginshared->mutex);
+ ginshared->nparticipantsdone++;
+ ginshared->reltuples += state->bs_reltuples;
+ ginshared->indtuples += state->bs_numtuples;
+ SpinLockRelease(&ginshared->mutex);
+
+ /* Notify leader */
+ ConditionVariableSignal(&ginshared->workersdonecv);
+
+ tuplesort_end(state->bs_sortstate);
+}
+
+/*
+ * Perform work within a launched parallel process.
+ */
+void
+_gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
+{
+ char *sharedquery;
+ GinShared *ginshared;
+ Sharedsort *sharedsort;
+ GinBuildState buildstate;
+ Relation heapRel;
+ Relation indexRel;
+ LOCKMODE heapLockmode;
+ LOCKMODE indexLockmode;
+ WalUsage *walusage;
+ BufferUsage *bufferusage;
+ int sortmem;
+
+ /*
+ * The only possible status flag that can be set to the parallel worker is
+ * PROC_IN_SAFE_IC.
+ */
+ Assert((MyProc->statusFlags == 0) ||
+ (MyProc->statusFlags == PROC_IN_SAFE_IC));
+
+ /* Set debug_query_string for individual workers first */
+ sharedquery = shm_toc_lookup(toc, PARALLEL_KEY_QUERY_TEXT, true);
+ debug_query_string = sharedquery;
+
+ /* Report the query string from leader */
+ pgstat_report_activity(STATE_RUNNING, debug_query_string);
+
+ /* Look up gin shared state */
+ ginshared = shm_toc_lookup(toc, PARALLEL_KEY_GIN_SHARED, false);
+
+ /* Open relations using lock modes known to be obtained by index.c */
+ if (!ginshared->isconcurrent)
+ {
+ heapLockmode = ShareLock;
+ indexLockmode = AccessExclusiveLock;
+ }
+ else
+ {
+ heapLockmode = ShareUpdateExclusiveLock;
+ indexLockmode = RowExclusiveLock;
+ }
+
+ /* Open relations within worker */
+ heapRel = table_open(ginshared->heaprelid, heapLockmode);
+ indexRel = index_open(ginshared->indexrelid, indexLockmode);
+
+ /* initialize the GIN build state */
+ initGinState(&buildstate.ginstate, indexRel);
+ buildstate.indtuples = 0;
+ memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
+
+ /*
+ * create a temporary memory context that is used to hold data not yet
+ * dumped out to the index
+ */
+ buildstate.tmpCtx = AllocSetContextCreate(CurrentMemoryContext,
+ "Gin build temporary context",
+ ALLOCSET_DEFAULT_SIZES);
+
+ /*
+ * create a temporary memory context that is used for calling
+ * ginExtractEntries(), and can be reset after each tuple
+ */
+ buildstate.funcCtx = AllocSetContextCreate(CurrentMemoryContext,
+ "Gin build temporary context for user-defined function",
+ ALLOCSET_DEFAULT_SIZES);
+
+ buildstate.accum.ginstate = &buildstate.ginstate;
+ ginInitBA(&buildstate.accum);
+
+
+ /* Look up shared state private to tuplesort.c */
+ sharedsort = shm_toc_lookup(toc, PARALLEL_KEY_TUPLESORT, false);
+ tuplesort_attach_shared(sharedsort, seg);
+
+ /* Prepare to track buffer usage during parallel execution */
+ InstrStartParallelQuery();
+
+ /*
+ * Might as well use reliable figure when doling out maintenance_work_mem
+ * (when requested number of workers were not launched, this will be
+ * somewhat higher than it is for other workers).
+ */
+ sortmem = maintenance_work_mem / ginshared->scantuplesortstates;
+
+ _gin_parallel_scan_and_build(&buildstate, ginshared, sharedsort,
+ heapRel, indexRel, sortmem, false);
+
+ /* Report WAL/buffer usage during parallel execution */
+ bufferusage = shm_toc_lookup(toc, PARALLEL_KEY_BUFFER_USAGE, false);
+ walusage = shm_toc_lookup(toc, PARALLEL_KEY_WAL_USAGE, false);
+ InstrEndParallelQuery(&bufferusage[ParallelWorkerNumber],
+ &walusage[ParallelWorkerNumber]);
+
+ index_close(indexRel, indexLockmode);
+ table_close(heapRel, heapLockmode);
+}
+
+/*
+ * _gin_build_tuple
+ * Serialize the state for an index key into a tuple for tuplesort.
+ *
+ * The tuple has a number of scalar fields (mostly matching the build state),
+ * and then a data array that stores the key first, and then the TID list.
+ *
+ * For by-reference data types, we store the actual data. For by-val types
+ * we simply copy the whole Datum, so that we don't have to care about stuff
+ * like endianess etc. We could make it a little bit smaller, but it's not
+ * worth it - it's a tiny fraction of the data, and we need to MAXALIGN the
+ * start of the TID list anyway. So we wouldn't save anything.
+ */
+static GinTuple *
+_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+ Datum key, int16 typlen, bool typbyval,
+ ItemPointerData *items, uint32 nitems,
+ Size *len)
+{
+ GinTuple *tuple;
+ char *ptr;
+
+ Size tuplen;
+ int keylen;
+
+ /*
+ * Calculate how long is the key value. Only keys with GIN_CAT_NORM_KEY
+ * have actual non-empty key. We include varlena headers and \0 bytes for
+ * strings, to make it easier to access the data in-line.
+ *
+ * For byval types we simply copy the whole Datum. We could store just the
+ * necessary bytes, but this is simpler to work with and not worth the
+ * extra complexity. Moreover we still need to do the MAXALIGN to allow
+ * direct access to items pointers.
+ */
+ if (category != GIN_CAT_NORM_KEY)
+ keylen = 0;
+ else if (typbyval)
+ keylen = sizeof(Datum);
+ else if (typlen > 0)
+ keylen = typlen;
+ else if (typlen == -1)
+ keylen = VARSIZE_ANY(key);
+ else if (typlen == -2)
+ keylen = strlen(DatumGetPointer(key)) + 1;
+ else
+ elog(ERROR, "invalid typlen");
+
+ /*
+ * Determine GIN tuple length with all the data included. Be careful about
+ * alignment, to allow direct access to item pointers.
+ */
+ tuplen = MAXALIGN(offsetof(GinTuple, data) + keylen) +
+ (sizeof(ItemPointerData) * nitems);
+
+ *len = tuplen;
+
+ /*
+ * allocate space for the whole GIN tuple
+ *
+ * XXX palloc0 so that valgrind does not complain about uninitialized
+ * bytes in writetup_index_gin, likely because of padding
+ */
+ tuple = palloc0(tuplen);
+
+ tuple->tuplen = tuplen;
+ tuple->attrnum = attrnum;
+ tuple->category = category;
+ tuple->keylen = keylen;
+ tuple->nitems = nitems;
+
+ /* key type info */
+ tuple->typlen = typlen;
+ tuple->typbyval = typbyval;
+
+ /*
+ * Copy the key and items into the tuple. First the key value, which we
+ * can simply copy right at the beginning of the data array.
+ */
+ if (category == GIN_CAT_NORM_KEY)
+ {
+ if (typbyval)
+ {
+ memcpy(tuple->data, &key, sizeof(Datum));
+ }
+ else if (typlen > 0) /* byref, fixed length */
+ {
+ memcpy(tuple->data, &key, typlen);
+ }
+ else if (typlen == -1)
+ {
+ memcpy(tuple->data, DatumGetPointer(key), keylen);
+ }
+ else if (typlen == -2)
+ {
+ memcpy(tuple->data, DatumGetPointer(key), keylen);
+ }
+ }
+
+ /* finally, copy the TIDs into the array */
+ ptr = (char *) tuple + MAXALIGN(offsetof(GinTuple, data) + keylen);
+
+ memcpy(ptr, items, sizeof(ItemPointerData) * nitems);
+
+ return tuple;
+}
+
+/*
+ * _gin_parse_tuple
+ * Deserialize the tuple from the tuplestore representation.
+ *
+ * Most of the fields are actually directly accessible, the only thing that
+ * needs more care is the key and the TID list.
+ *
+ * For the key, this returns a regular Datum representing it. It's either the
+ * actual key value, or a pointer to the beginning of the data array (which is
+ * where the data was copied by _gin_build_tuple).
+ *
+ * The pointer to the TID list is returned through 'items' (which is simply
+ * a pointer to the data array).
+ */
+static Datum
+_gin_parse_tuple(GinTuple *a, ItemPointerData **items)
+{
+ Datum key;
+
+ if (items)
+ {
+ char *ptr = (char *) a + MAXALIGN(offsetof(GinTuple, data) + a->keylen);
+
+ *items = (ItemPointerData *) ptr;
+ }
+
+ if (a->category != GIN_CAT_NORM_KEY)
+ return (Datum) 0;
+
+ if (a->typbyval > 0)
+ {
+ memcpy(&key, a->data, a->keylen);
+ return key;
+ }
+
+ return PointerGetDatum(a->data);
+}
+
+/*
+ * _gin_compare_tuples
+ * Compare GIN tuples, used by tuplesort during parallel index build.
+ *
+ * The scalar fields (attrnum, category) are compared first, the key value is
+ * compared last. The comparisons are done simply by "memcmp", based on the
+ * assumption that if we get two keys that are two different representations
+ * of a logically equal value, it'll get merged by the index build.
+ *
+ * FIXME Is the assumption we can just memcmp() actually valid? Won't this
+ * trigger the "could not split GIN page; all old items didn't fit" error
+ * when trying to update the TID list?
+ */
+int
+_gin_compare_tuples(GinTuple *a, GinTuple *b)
+{
+ Datum keya,
+ keyb;
+
+ if (a->attrnum < b->attrnum)
+ return -1;
+
+ if (a->attrnum > b->attrnum)
+ return 1;
+
+ if (a->category < b->category)
+ return -1;
+
+ if (a->category > b->category)
+ return 1;
+
+ if ((a->category == GIN_CAT_NORM_KEY) &&
+ (b->category == GIN_CAT_NORM_KEY))
+ {
+ keya = _gin_parse_tuple(a, NULL);
+ keyb = _gin_parse_tuple(b, NULL);
+
+ if (a->typlen > 0)
+ return memcmp(&keya, &keyb, a->keylen);
+
+ if (a->typlen < 0)
+ {
+ if (a->keylen < b->keylen)
+ return -1;
+
+ if (a->keylen > b->keylen)
+ return 1;
+
+ return memcmp(DatumGetPointer(keya), DatumGetPointer(keyb), a->keylen);
+ }
+ }
+
+ return 0;
+}
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 5747ae6a4ca..dd22b44aca9 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -53,7 +53,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->amclusterable = false;
amroutine->ampredlocks = true;
amroutine->amcanparallel = false;
- amroutine->amcanbuildparallel = false;
+ amroutine->amcanbuildparallel = true;
amroutine->amcaninclude = false;
amroutine->amusemaintenanceworkmem = true;
amroutine->amsummarizing = false;
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index 8613fc6fb54..c9ea769afb5 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/brin.h"
+#include "access/gin.h"
#include "access/nbtree.h"
#include "access/parallel.h"
#include "access/session.h"
@@ -146,6 +147,9 @@ static const struct
{
"_brin_parallel_build_main", _brin_parallel_build_main
},
+ {
+ "_gin_parallel_build_main", _gin_parallel_build_main
+ },
{
"parallel_vacuum_main", parallel_vacuum_main
}
diff --git a/src/backend/utils/sort/tuplesortvariants.c b/src/backend/utils/sort/tuplesortvariants.c
index 05a853caa36..55cc55969e5 100644
--- a/src/backend/utils/sort/tuplesortvariants.c
+++ b/src/backend/utils/sort/tuplesortvariants.c
@@ -20,6 +20,7 @@
#include "postgres.h"
#include "access/brin_tuple.h"
+#include "access/gin_tuple.h"
#include "access/hash.h"
#include "access/htup_details.h"
#include "access/nbtree.h"
@@ -46,6 +47,8 @@ static void removeabbrev_index(Tuplesortstate *state, SortTuple *stups,
int count);
static void removeabbrev_index_brin(Tuplesortstate *state, SortTuple *stups,
int count);
+static void removeabbrev_index_gin(Tuplesortstate *state, SortTuple *stups,
+ int count);
static void removeabbrev_datum(Tuplesortstate *state, SortTuple *stups,
int count);
static int comparetup_heap(const SortTuple *a, const SortTuple *b,
@@ -74,6 +77,8 @@ static int comparetup_index_hash_tiebreak(const SortTuple *a, const SortTuple *b
Tuplesortstate *state);
static int comparetup_index_brin(const SortTuple *a, const SortTuple *b,
Tuplesortstate *state);
+static int comparetup_index_gin(const SortTuple *a, const SortTuple *b,
+ Tuplesortstate *state);
static void writetup_index(Tuplesortstate *state, LogicalTape *tape,
SortTuple *stup);
static void readtup_index(Tuplesortstate *state, SortTuple *stup,
@@ -82,6 +87,10 @@ static void writetup_index_brin(Tuplesortstate *state, LogicalTape *tape,
SortTuple *stup);
static void readtup_index_brin(Tuplesortstate *state, SortTuple *stup,
LogicalTape *tape, unsigned int len);
+static void writetup_index_gin(Tuplesortstate *state, LogicalTape *tape,
+ SortTuple *stup);
+static void readtup_index_gin(Tuplesortstate *state, SortTuple *stup,
+ LogicalTape *tape, unsigned int len);
static int comparetup_datum(const SortTuple *a, const SortTuple *b,
Tuplesortstate *state);
static int comparetup_datum_tiebreak(const SortTuple *a, const SortTuple *b,
@@ -580,6 +589,35 @@ tuplesort_begin_index_brin(int workMem,
return state;
}
+
+Tuplesortstate *
+tuplesort_begin_index_gin(int workMem, SortCoordinate coordinate,
+ int sortopt)
+{
+ Tuplesortstate *state = tuplesort_begin_common(workMem, coordinate,
+ sortopt);
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+
+#ifdef TRACE_SORT
+ if (trace_sort)
+ elog(LOG,
+ "begin index sort: workMem = %d, randomAccess = %c",
+ workMem,
+ sortopt & TUPLESORT_RANDOMACCESS ? 't' : 'f');
+#endif
+
+ base->nKeys = 1; /* Only the index key */
+
+ base->removeabbrev = removeabbrev_index_gin;
+ base->comparetup = comparetup_index_gin;
+ base->writetup = writetup_index_gin;
+ base->readtup = readtup_index_gin;
+ base->haveDatum1 = false;
+ base->arg = NULL;
+
+ return state;
+}
+
Tuplesortstate *
tuplesort_begin_datum(Oid datumType, Oid sortOperator, Oid sortCollation,
bool nullsFirstFlag, int workMem,
@@ -817,6 +855,37 @@ tuplesort_putbrintuple(Tuplesortstate *state, BrinTuple *tuple, Size size)
MemoryContextSwitchTo(oldcontext);
}
+void
+tuplesort_putgintuple(Tuplesortstate *state, GinTuple *tup, Size size)
+{
+ SortTuple stup;
+ GinTuple *ctup;
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ MemoryContext oldcontext = MemoryContextSwitchTo(base->tuplecontext);
+ Size tuplen;
+
+ /* copy the GinTuple into the right memory context */
+ ctup = palloc(size);
+ memcpy(ctup, tup, size);
+
+ stup.tuple = ctup;
+ stup.datum1 = (Datum) 0;
+ stup.isnull1 = false;
+
+ /* GetMemoryChunkSpace is not supported for bump contexts */
+ if (TupleSortUseBumpTupleCxt(base->sortopt))
+ tuplen = MAXALIGN(size);
+ else
+ tuplen = GetMemoryChunkSpace(ctup);
+
+ tuplesort_puttuple_common(state, &stup,
+ base->sortKeys &&
+ base->sortKeys->abbrev_converter &&
+ !stup.isnull1, tuplen);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Accept one Datum while collecting input data for sort.
*
@@ -989,6 +1058,29 @@ tuplesort_getbrintuple(Tuplesortstate *state, Size *len, bool forward)
return &btup->tuple;
}
+GinTuple *
+tuplesort_getgintuple(Tuplesortstate *state, Size *len, bool forward)
+{
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ MemoryContext oldcontext = MemoryContextSwitchTo(base->sortcontext);
+ SortTuple stup;
+ GinTuple *tup;
+
+ if (!tuplesort_gettuple_common(state, forward, &stup))
+ stup.tuple = NULL;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ if (!stup.tuple)
+ return false;
+
+ tup = (GinTuple *) stup.tuple;
+
+ *len = tup->tuplen;
+
+ return tup;
+}
+
/*
* Fetch the next Datum in either forward or back direction.
* Returns false if no more datums.
@@ -1777,6 +1869,68 @@ readtup_index_brin(Tuplesortstate *state, SortTuple *stup,
stup->datum1 = tuple->tuple.bt_blkno;
}
+
+/*
+ * Routines specialized for GIN case
+ */
+
+static void
+removeabbrev_index_gin(Tuplesortstate *state, SortTuple *stups, int count)
+{
+ Assert(false);
+ elog(ERROR, "removeabbrev_index_gin not implemented");
+}
+
+static int
+comparetup_index_gin(const SortTuple *a, const SortTuple *b,
+ Tuplesortstate *state)
+{
+ Assert(!TuplesortstateGetPublic(state)->haveDatum1);
+
+ return _gin_compare_tuples((GinTuple *) a->tuple,
+ (GinTuple *) b->tuple);
+}
+
+static void
+writetup_index_gin(Tuplesortstate *state, LogicalTape *tape, SortTuple *stup)
+{
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ GinTuple *tuple = (GinTuple *) stup->tuple;
+ unsigned int tuplen = tuple->tuplen;
+
+ tuplen = tuplen + sizeof(tuplen);
+ LogicalTapeWrite(tape, &tuplen, sizeof(tuplen));
+ LogicalTapeWrite(tape, tuple, tuple->tuplen);
+ if (base->sortopt & TUPLESORT_RANDOMACCESS) /* need trailing length word? */
+ LogicalTapeWrite(tape, &tuplen, sizeof(tuplen));
+}
+
+static void
+readtup_index_gin(Tuplesortstate *state, SortTuple *stup,
+ LogicalTape *tape, unsigned int len)
+{
+ GinTuple *tuple;
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ unsigned int tuplen = len - sizeof(unsigned int);
+
+ /*
+ * Allocate space for the GIN sort tuple, which already has the proper
+ * length included in the header.
+ */
+ tuple = (GinTuple *) tuplesort_readtup_alloc(state, tuplen);
+
+ tuple->tuplen = tuplen;
+
+ LogicalTapeReadExact(tape, tuple, tuplen);
+ if (base->sortopt & TUPLESORT_RANDOMACCESS) /* need trailing length word? */
+ LogicalTapeReadExact(tape, &tuplen, sizeof(tuplen));
+ stup->tuple = (void *) tuple;
+
+ /* no abbreviations (FIXME maybe use attrnum for this?) */
+ stup->datum1 = (Datum) 0;
+}
+
+
/*
* Routines specialized for DatumTuple case
*/
diff --git a/src/include/access/gin.h b/src/include/access/gin.h
index 25983b7a505..be76d8446f4 100644
--- a/src/include/access/gin.h
+++ b/src/include/access/gin.h
@@ -12,6 +12,8 @@
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
+#include "nodes/execnodes.h"
+#include "storage/shm_toc.h"
#include "storage/block.h"
#include "utils/relcache.h"
@@ -88,4 +90,6 @@ extern void ginGetStats(Relation index, GinStatsData *stats);
extern void ginUpdateStats(Relation index, const GinStatsData *stats,
bool is_build);
+extern void _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc);
+
#endif /* GIN_H */
diff --git a/src/include/access/gin_tuple.h b/src/include/access/gin_tuple.h
new file mode 100644
index 00000000000..b5304b73ff1
--- /dev/null
+++ b/src/include/access/gin_tuple.h
@@ -0,0 +1,28 @@
+/*--------------------------------------------------------------------------
+ * gin.h
+ * Public header file for Generalized Inverted Index access method.
+ *
+ * Copyright (c) 2006-2024, PostgreSQL Global Development Group
+ *
+ * src/include/access/gin.h
+ *--------------------------------------------------------------------------
+ */
+#ifndef GIN_TUPLE_
+#define GIN_TUPLE_
+
+
+typedef struct GinTuple
+{
+ Size tuplen; /* length of the whole tuple */
+ Size keylen; /* bytes in data for key value */
+ int16 typlen; /* typlen for key */
+ bool typbyval; /* typbyval for key */
+ OffsetNumber attrnum;
+ signed char category; /* category: normal or NULL? */
+ int nitems; /* number of TIDs in the data */
+ char data[FLEXIBLE_ARRAY_MEMBER];
+} GinTuple;
+
+extern int _gin_compare_tuples(GinTuple *a, GinTuple *b);
+
+#endif /* GIN_TUPLE_H */
diff --git a/src/include/utils/tuplesort.h b/src/include/utils/tuplesort.h
index e7941a1f09f..35fa5ae2442 100644
--- a/src/include/utils/tuplesort.h
+++ b/src/include/utils/tuplesort.h
@@ -22,6 +22,7 @@
#define TUPLESORT_H
#include "access/brin_tuple.h"
+#include "access/gin_tuple.h"
#include "access/itup.h"
#include "executor/tuptable.h"
#include "storage/dsm.h"
@@ -443,6 +444,8 @@ extern Tuplesortstate *tuplesort_begin_index_gist(Relation heapRel,
int sortopt);
extern Tuplesortstate *tuplesort_begin_index_brin(int workMem, SortCoordinate coordinate,
int sortopt);
+extern Tuplesortstate *tuplesort_begin_index_gin(int workMem, SortCoordinate coordinate,
+ int sortopt);
extern Tuplesortstate *tuplesort_begin_datum(Oid datumType,
Oid sortOperator, Oid sortCollation,
bool nullsFirstFlag,
@@ -456,6 +459,7 @@ extern void tuplesort_putindextuplevalues(Tuplesortstate *state,
Relation rel, ItemPointer self,
const Datum *values, const bool *isnull);
extern void tuplesort_putbrintuple(Tuplesortstate *state, BrinTuple *tup, Size len);
+extern void tuplesort_putgintuple(Tuplesortstate *state, GinTuple *tup, Size len);
extern void tuplesort_putdatum(Tuplesortstate *state, Datum val,
bool isNull);
@@ -465,6 +469,8 @@ extern HeapTuple tuplesort_getheaptuple(Tuplesortstate *state, bool forward);
extern IndexTuple tuplesort_getindextuple(Tuplesortstate *state, bool forward);
extern BrinTuple *tuplesort_getbrintuple(Tuplesortstate *state, Size *len,
bool forward);
+extern GinTuple *tuplesort_getgintuple(Tuplesortstate *state, Size *len,
+ bool forward);
extern bool tuplesort_getdatum(Tuplesortstate *state, bool forward, bool copy,
Datum *val, bool *isNull, Datum *abbrev);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e10ff28ee54..0ee1b9ab19b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1004,11 +1004,13 @@ GinBtreeData
GinBtreeDataLeafInsertData
GinBtreeEntryInsertData
GinBtreeStack
+GinBuffer
GinBuildState
GinChkVal
GinEntries
GinEntryAccumulator
GinIndexStat
+GinLeader
GinMetaPageData
GinNullCategory
GinOptions
@@ -1021,9 +1023,11 @@ GinScanEntry
GinScanKey
GinScanOpaque
GinScanOpaqueData
+GinShared
GinState
GinStatsData
GinTernaryValue
+GinTuple
GinTupleCollector
GinVacuumState
GistBuildMode
--
2.44.0
[text/x-patch] v20240502-0002-Use-mergesort-in-the-leader-process.patch (9.6K, ../../[email protected]/5-v20240502-0002-Use-mergesort-in-the-leader-process.patch)
download | inline diff:
From bc51aa8ff10b0a53f65564ace2a14da671363341 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:32 +0200
Subject: [PATCH v20240502 2/8] Use mergesort in the leader process
The leader process (executing the serial part of the index build) spent
a significant part of the time in pg_qsort, after combining the partial
results from the workers. But we can improve this and move some of the
costs to the parallel part in workers - if workers produce sorted TID
lists, the leader can combine them by mergesort.
But to make this really efficient, the mergesort must not be executed
too many times. The workers may easily produce very short TID lists, if
there are many different keys, hitting the memory limit often. So this
adds an intermediate tuplesort pass into each worker, to combine TIDs
for each key and only then write the result into the shared tuplestore.
This means the number of mergesort invocations for each key should be
about the same as the number of workers. We can't really do better, and
it's low enough to keep the mergesort approach efficient.
Note: If we introduce a memory limit on GinBuffer (to not accumulate too
many TIDs in memory), we could end up with more chunks, but it should
not be very common.
---
src/backend/access/gin/gininsert.c | 171 +++++++++++++++++++++++++----
1 file changed, 148 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 4d6d152403e..8011e0b5ad5 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -161,6 +161,14 @@ typedef struct
* build callback etc.
*/
Tuplesortstate *bs_sortstate;
+
+ /*
+ * The sortstate is used only within a worker for the first merge pass
+ * that happens in the worker. In principle it doesn't need to be part of
+ * the build state and we could pass it around directly, but it's more
+ * convenient this way.
+ */
+ Tuplesortstate *bs_worker_sort;
} GinBuildState;
@@ -533,7 +541,7 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
key, attr->attlen, attr->attbyval,
list, nlist, &tuplen);
- tuplesort_putgintuple(buildstate->bs_sortstate, tup, tuplen);
+ tuplesort_putgintuple(buildstate->bs_worker_sort, tup, tuplen);
pfree(tup);
}
@@ -1127,7 +1135,6 @@ typedef struct GinBuffer
/* array of TID values */
int nitems;
- int maxitems;
ItemPointerData *items;
} GinBuffer;
@@ -1136,7 +1143,6 @@ static void
AssertCheckGinBuffer(GinBuffer *buffer)
{
#ifdef USE_ASSERT_CHECKING
- Assert(buffer->nitems <= buffer->maxitems);
#endif
}
@@ -1240,28 +1246,22 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->key = (Datum) 0;
}
- /* enlarge the TID buffer, if needed */
- if (buffer->nitems + tup->nitems > buffer->maxitems)
+ /* copy the new TIDs into the buffer, combine using merge-sort */
{
- /* 64 seems like a good init value */
- buffer->maxitems = Max(buffer->maxitems, 64);
+ int nnew;
+ ItemPointer new;
- while (buffer->nitems + tup->nitems > buffer->maxitems)
- buffer->maxitems *= 2;
+ new = ginMergeItemPointers(buffer->items, buffer->nitems,
+ items, tup->nitems, &nnew);
- if (buffer->items == NULL)
- buffer->items = palloc(buffer->maxitems * sizeof(ItemPointerData));
- else
- buffer->items = repalloc(buffer->items,
- buffer->maxitems * sizeof(ItemPointerData));
- }
+ Assert(nnew == buffer->nitems + tup->nitems);
- /* now we should be guaranteed to have enough space for all the TIDs */
- Assert(buffer->nitems + tup->nitems <= buffer->maxitems);
+ if (buffer->items)
+ pfree(buffer->items);
- /* copy the new TIDs into the buffer */
- memcpy(&buffer->items[buffer->nitems], items, sizeof(ItemPointerData) * tup->nitems);
- buffer->nitems += tup->nitems;
+ buffer->items = new;
+ buffer->nitems = nnew;
+ }
AssertCheckItemPointers(buffer->items, buffer->nitems, false);
}
@@ -1302,6 +1302,21 @@ GinBufferReset(GinBuffer *buffer)
/* XXX should do something with extremely large array of items? */
}
+/* XXX probably would be better to have a memory context for the buffer */
+static void
+GinBufferFree(GinBuffer *buffer)
+{
+ if (buffer->items)
+ pfree(buffer->items);
+
+ /* release byref values, do nothing for by-val ones */
+ if (!GinBufferIsEmpty(buffer) &&
+ (buffer->category == GIN_CAT_NORM_KEY) && !buffer->typbyval)
+ pfree(DatumGetPointer(buffer->key));
+
+ pfree(buffer);
+}
+
/*
* XXX Maybe check size of the TID arrays, and return false if it's too
* large (more thant maintenance_work_mem or something?).
@@ -1375,7 +1390,7 @@ _gin_parallel_merge(GinBuildState *state)
* the data into the insert, and start a new entry for current
* GinTuple.
*/
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer->items, buffer->nitems, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1392,7 +1407,7 @@ _gin_parallel_merge(GinBuildState *state)
/* flush data remaining in the buffer (for the last key) */
if (!GinBufferIsEmpty(buffer))
{
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer->items, buffer->nitems, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1402,6 +1417,9 @@ _gin_parallel_merge(GinBuildState *state)
GinBufferReset(buffer);
}
+ /* relase all the memory */
+ GinBufferFree(buffer);
+
tuplesort_end(state->bs_sortstate);
return reltuples;
@@ -1440,6 +1458,102 @@ _gin_leader_participate_as_worker(GinBuildState *buildstate, Relation heap, Rela
ginleader->sharedsort, heap, index, sortmem, true);
}
+/*
+ * _gin_process_worker_data
+ * First phase of the key merging, happening in the worker.
+ *
+ * Depending on the number of distinct keys, the TID lists produced by the
+ * callback may be very short. But combining many tiny lists is expensive,
+ * so we try to do as much as possible in the workers and only then pass the
+ * results to the leader.
+ *
+ * We read the tuples sorted by the key, and merge them into larger lists.
+ * At the moment there's no memory limit, so this will just produce one
+ * huge (sorted) list per key in each worker. Which means the leader will
+ * do a very limited number of mergesorts, which is good.
+ */
+static void
+_gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
+{
+ GinTuple *tup;
+ Size tuplen;
+
+ GinBuffer *buffer;
+
+ /* initialize buffer to combine entries for the same key */
+ buffer = GinBufferInit();
+
+ /* sort the raw per-worker data */
+ tuplesort_performsort(state->bs_worker_sort);
+
+ /*
+ * Read the GIN tuples from the shared tuplesort, sorted by the key, and
+ * merge them into larger chunks for the leader to combine.
+ */
+ while ((tup = tuplesort_getgintuple(worker_sort, &tuplen, true)) != NULL)
+ {
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * If the buffer can accept the new GIN tuple, just store it there and
+ * we're done. If it's a different key (or maybe too much data) flush
+ * the current contents into the index first.
+ */
+ if (!GinBufferCanAddKey(buffer, tup))
+ {
+ GinTuple *ntup;
+ Size ntuplen;
+
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ GinBufferSortItems(buffer);
+
+ ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ buffer->key, buffer->typlen, buffer->typbyval,
+ buffer->items, buffer->nitems, &ntuplen);
+
+ tuplesort_putgintuple(state->bs_sortstate, ntup, ntuplen);
+
+ pfree(ntup);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ /* now remember the new key */
+ GinBufferStoreTuple(buffer, tup);
+ }
+
+ /* flush data remaining in the buffer (for the last key) */
+ if (!GinBufferIsEmpty(buffer))
+ {
+ GinTuple *ntup;
+ Size ntuplen;
+
+ GinBufferSortItems(buffer);
+
+ ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ buffer->key, buffer->typlen, buffer->typbyval,
+ buffer->items, buffer->nitems, &ntuplen);
+
+ tuplesort_putgintuple(state->bs_sortstate, ntup, ntuplen);
+
+ pfree(ntup);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ /* relase all the memory */
+ GinBufferFree(buffer);
+
+ tuplesort_end(worker_sort);
+}
+
/*
* Perform a worker's portion of a parallel sort.
*
@@ -1471,6 +1585,10 @@ _gin_parallel_scan_and_build(GinBuildState *state,
state->bs_sortstate = tuplesort_begin_index_gin(sortmem, coordinate,
TUPLESORT_NONE);
+ /* Local per-worker sort of raw-data */
+ state->bs_worker_sort = tuplesort_begin_index_gin(sortmem, NULL,
+ TUPLESORT_NONE);
+
/* Join parallel scan */
indexInfo = BuildIndexInfo(index);
indexInfo->ii_Concurrent = ginshared->isconcurrent;
@@ -1508,7 +1626,7 @@ _gin_parallel_scan_and_build(GinBuildState *state,
key, attr->attlen, attr->attbyval,
list, nlist, &len);
- tuplesort_putgintuple(state->bs_sortstate, tup, len);
+ tuplesort_putgintuple(state->bs_worker_sort, tup, len);
pfree(tup);
}
@@ -1517,6 +1635,13 @@ _gin_parallel_scan_and_build(GinBuildState *state,
ginInitBA(&state->accum);
}
+ /*
+ * Do the first phase of in-worker processing - sort the data produced by
+ * the callback, and combine them into much larger chunks and place that
+ * into the shared tuplestore for leader to process.
+ */
+ _gin_process_worker_data(state, state->bs_worker_sort);
+
/* sort the GIN tuples built by this worker */
tuplesort_performsort(state->bs_sortstate);
--
2.44.0
[text/x-patch] v20240502-0003-Remove-the-explicit-pg_qsort-in-workers.patch (11.1K, ../../[email protected]/6-v20240502-0003-Remove-the-explicit-pg_qsort-in-workers.patch)
download | inline diff:
From c1dafb0209ce225e9e825c07ddcb1416ce2435fe Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:36 +0200
Subject: [PATCH v20240502 3/8] Remove the explicit pg_qsort in workers
We don't need to do the explicit sort before building the GIN tuple,
because the mergesort in GinBufferStoreTuple is already maintaining the
correct order (this was added in an earlier commit).
The comment also adds a field with the first TID, and modifies the
comparator to sort by it (for each key value). This helps workers to
build non-overlapping TID lists and simply append values instead of
having to do the actual mergesort to combine them. This is best-effort,
i.e. it's not guaranteed to eliminate the mergesort - in particular,
parallel scans are synchronized, and thus may start somewhere in the
middle of the table, and wrap around. In which case there may be very
wide list (with low/high TID values).
Note: There's an XXX comment with a couple ideas on how to improve this,
at the cost of more complexity.
---
src/backend/access/gin/gininsert.c | 130 +++++++++++++++++++++--------
src/include/access/gin_tuple.h | 9 +-
2 files changed, 104 insertions(+), 35 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 8011e0b5ad5..5762c9520d8 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -1110,12 +1110,6 @@ _gin_parallel_heapscan(GinBuildState *state)
return state->bs_reltuples;
}
-static int
-tid_cmp(const void *a, const void *b)
-{
- return ItemPointerCompare((ItemPointer) a, (ItemPointer) b);
-}
-
/*
* State used to combine accumulate TIDs from multiple GinTuples for the same
* key value.
@@ -1147,17 +1141,21 @@ AssertCheckGinBuffer(GinBuffer *buffer)
}
static void
-AssertCheckItemPointers(ItemPointerData *items, int nitems, bool sorted)
+AssertCheckItemPointers(GinBuffer *buffer, bool sorted)
{
#ifdef USE_ASSERT_CHECKING
- for (int i = 0; i < nitems; i++)
+ /* we should not have a buffer with no TIDs to sort */
+ Assert(buffer->items != NULL);
+ Assert(buffer->nitems > 0);
+
+ for (int i = 0; i < buffer->nitems; i++)
{
- Assert(ItemPointerIsValid(&items[i]));
+ Assert(ItemPointerIsValid(&buffer->items[i]));
if ((i == 0) || !sorted)
continue;
- Assert(ItemPointerCompare(&items[i - 1], &items[i]) < 0);
+ Assert(ItemPointerCompare(&buffer->items[i - 1], &buffer->items[i]) < 0);
}
#endif
}
@@ -1220,6 +1218,45 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
return (memcmp(tup->data, DatumGetPointer(buffer->key), buffer->keylen) == 0);
}
+/*
+ * GinBufferStoreTuple
+ * Add data from a GinTuple into the GinBuffer.
+ *
+ * If the buffer is empty, we simply initialize it with data from the tuple.
+ * Otherwise data from the tuple (the TID list) is added to the TID data in
+ * the buffer, either by simply appending the TIDs or doing merge sort.
+ *
+ * The data (for the same key) is expected to be processed sorted by first
+ * TID. But this does not guarantee the lists do not overlap, especially in
+ * the leader, because the workers process interleaving data. But even in
+ * a single worker, lists can overlap - parallel scans require sync-scans,
+ * and if the scan starts somewhere in the table and then wraps around, it
+ * may contain very wide lists (in terms of TID range).
+ *
+ * But the ginMergeItemPointers() is already smart about detecting cases
+ * where it can simply concatenate the lists, and when full mergesort is
+ * needed. And does the right thing.
+ *
+ * By keeping the first TID in the GinTuple and sorting by that, we make
+ * it more likely the lists won't overlap very often.
+ *
+ * XXX How frequent can the overlaps be? If the scan does not wrap around,
+ * there should be no overlapping lists, and thus no mergesort. After ab
+ * overlap, there probably can be many - the one list will be very wide,
+ * with a very low and high TID, and all other lists will overlap with it.
+ * I'm not sure how much we can do to prevent that, short of disabling sync
+ * scans (which for parallel scans is currently not possible). One option
+ * would be to keep two lists of TIDs, and see if the new list can be
+ * concatenated with one of them. The idea is that there's only one wide
+ * list (because the wraparound happens only once), and then do the
+ * mergesort only once at the very end.
+ *
+ * XXX Alternatively, we could simply detect the case when the lists can't
+ * be appended, and flush the current list out. The wrap around happens only
+ * once, so there's going to be only such wide list, and it'll be sorted
+ * first (because it has the lowest TID for the key). So we'd do this at
+ * most once per key.
+ */
static void
GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
{
@@ -1246,7 +1283,12 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->key = (Datum) 0;
}
- /* copy the new TIDs into the buffer, combine using merge-sort */
+ /*
+ * Copy the new TIDs into the buffer, combine with existing data (if any)
+ * using merge-sort. The mergesort is already smart about cases where it
+ * can simply concatenate the two lists, and when it actually needs to
+ * merge the data in an expensive way.
+ */
{
int nnew;
ItemPointer new;
@@ -1261,21 +1303,9 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->items = new;
buffer->nitems = nnew;
- }
-
- AssertCheckItemPointers(buffer->items, buffer->nitems, false);
-}
-
-static void
-GinBufferSortItems(GinBuffer *buffer)
-{
- /* we should not have a buffer with no TIDs to sort */
- Assert(buffer->items != NULL);
- Assert(buffer->nitems > 0);
-
- pg_qsort(buffer->items, buffer->nitems, sizeof(ItemPointerData), tid_cmp);
- AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+ AssertCheckItemPointers(buffer, true);
+ }
}
/* XXX probably would be better to have a memory context for the buffer */
@@ -1299,6 +1329,11 @@ GinBufferReset(GinBuffer *buffer)
buffer->typlen = 0;
buffer->typbyval = 0;
+ if (buffer->items)
+ {
+ pfree(buffer->items);
+ buffer->items = NULL;
+ }
/* XXX should do something with extremely large array of items? */
}
@@ -1390,7 +1425,7 @@ _gin_parallel_merge(GinBuildState *state)
* the data into the insert, and start a new entry for current
* GinTuple.
*/
- AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+ AssertCheckItemPointers(buffer, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1400,14 +1435,17 @@ _gin_parallel_merge(GinBuildState *state)
GinBufferReset(buffer);
}
- /* now remember the new key */
+ /*
+ * Remember data for the current tuple (either remember the new key,
+ * or append if to the existing data).
+ */
GinBufferStoreTuple(buffer, tup);
}
/* flush data remaining in the buffer (for the last key) */
if (!GinBufferIsEmpty(buffer))
{
- AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+ AssertCheckItemPointers(buffer, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1510,7 +1548,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
* the data into the insert, and start a new entry for current
* GinTuple.
*/
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer, true);
ntup = _gin_build_tuple(buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
@@ -1524,7 +1562,10 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinBufferReset(buffer);
}
- /* now remember the new key */
+ /*
+ * Remember data for the current tuple (either remember the new key,
+ * or append if to the existing data).
+ */
GinBufferStoreTuple(buffer, tup);
}
@@ -1534,7 +1575,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinTuple *ntup;
Size ntuplen;
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer, true);
ntup = _gin_build_tuple(buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
@@ -1835,6 +1876,7 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
tuple->category = category;
tuple->keylen = keylen;
tuple->nitems = nitems;
+ tuple->first = items[0];
/* key type info */
tuple->typlen = typlen;
@@ -1919,6 +1961,12 @@ _gin_parse_tuple(GinTuple *a, ItemPointerData **items)
* assumption that if we get two keys that are two different representations
* of a logically equal value, it'll get merged by the index build.
*
+ * If the key value matches, we compare the first TID value in the TID list,
+ * which means the tuples are merged in an order in which they are most
+ * likely to be simply concatenated. (This "first" TID will also allow us
+ * to determine a point up to which the list is fully determined and can be
+ * written into the index to enforce a memory limit etc.)
+ *
* FIXME Is the assumption we can just memcmp() actually valid? Won't this
* trigger the "could not split GIN page; all old items didn't fit" error
* when trying to update the TID list?
@@ -1947,20 +1995,34 @@ _gin_compare_tuples(GinTuple *a, GinTuple *b)
keya = _gin_parse_tuple(a, NULL);
keyb = _gin_parse_tuple(b, NULL);
+ /*
+ * works for both byval and byref types with fixed lenght, because for
+ * byval we set keylen to sizeof(Datum)
+ */
if (a->typlen > 0)
- return memcmp(&keya, &keyb, a->keylen);
+ {
+ int r = memcmp(&keya, &keyb, a->keylen);
+
+ /* if the key is the same, consider the first TID in the array */
+ return (r != 0) ? r : ItemPointerCompare(&a->first, &b->first);
+ }
if (a->typlen < 0)
{
+ int r;
+
if (a->keylen < b->keylen)
return -1;
if (a->keylen > b->keylen)
return 1;
- return memcmp(DatumGetPointer(keya), DatumGetPointer(keyb), a->keylen);
+ r = memcmp(DatumGetPointer(keya), DatumGetPointer(keyb), a->keylen);
+
+ /* if the key is the same, consider the first TID in the array */
+ return (r != 0) ? r : ItemPointerCompare(&a->first, &b->first);
}
}
- return 0;
+ return ItemPointerCompare(&a->first, &b->first);
}
diff --git a/src/include/access/gin_tuple.h b/src/include/access/gin_tuple.h
index b5304b73ff1..c3641edd5fc 100644
--- a/src/include/access/gin_tuple.h
+++ b/src/include/access/gin_tuple.h
@@ -10,7 +10,13 @@
#ifndef GIN_TUPLE_
#define GIN_TUPLE_
-
+/*
+ * Each worker sees tuples in CTID order, so if we track the first TID and
+ * compare that when combining results in the worker, we would not need to
+ * do an expensive sort in workers (the mergesort is already smart about
+ * detecting this and just concatenating the lists). We'd still need the
+ * full mergesort in the leader, but that's much cheaper.
+ */
typedef struct GinTuple
{
Size tuplen; /* length of the whole tuple */
@@ -19,6 +25,7 @@ typedef struct GinTuple
bool typbyval; /* typbyval for key */
OffsetNumber attrnum;
signed char category; /* category: normal or NULL? */
+ ItemPointerData first; /* first TID in the array */
int nitems; /* number of TIDs in the data */
char data[FLEXIBLE_ARRAY_MEMBER];
} GinTuple;
--
2.44.0
[text/x-patch] v20240502-0004-Compress-TID-lists-before-writing-tuples-t.patch (8.0K, ../../[email protected]/7-v20240502-0004-Compress-TID-lists-before-writing-tuples-t.patch)
download | inline diff:
From 6e6fc47199850c55069bf380079f61029f5a1b66 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:39 +0200
Subject: [PATCH v20240502 4/8] Compress TID lists before writing tuples to
disk
When serializing GIN tuples to tuplesorts, we can significantly reduce
the amount of data by compressing the TID lists. The GIN opclasses may
produce a lot of data (depending on how many keys are extracted from
each row), and the compression is very effective and efficient.
If the number of different keys is high, the first worker pass may not
benefit from the compression very much - the data will be spilled to
disk before the TID lists can grow long enough for the compression to
actualy help. In the second pass the impact is much more significant.
For real-world data (full-text on mailing list archives), I usually see
the compression to save only about ~15% in the first pass, but ~50% on
the second pass.
---
src/backend/access/gin/gininsert.c | 116 +++++++++++++++++++++++------
src/tools/pgindent/typedefs.list | 1 +
2 files changed, 95 insertions(+), 22 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 5762c9520d8..607ce9b34d6 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -187,7 +187,9 @@ static void _gin_parallel_scan_and_build(GinBuildState *buildstate,
Relation heap, Relation index,
int sortmem, bool progress);
-static Datum _gin_parse_tuple(GinTuple *a, ItemPointerData **items);
+static ItemPointer _gin_parse_tuple_items(GinTuple *a);
+static Datum _gin_parse_tuple_key(GinTuple *a);
+
static GinTuple *_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
Datum key, int16 typlen, bool typbyval,
ItemPointerData *items, uint32 nitems,
@@ -1265,7 +1267,8 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
AssertCheckGinBuffer(buffer);
- key = _gin_parse_tuple(tup, &items);
+ key = _gin_parse_tuple_key(tup);
+ items = _gin_parse_tuple_items(tup);
/* if the buffer is empty, set the fields (and copy the key) */
if (GinBufferIsEmpty(buffer))
@@ -1306,6 +1309,9 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
AssertCheckItemPointers(buffer, true);
}
+
+ /* free the decompressed TID list */
+ pfree(items);
}
/* XXX probably would be better to have a memory context for the buffer */
@@ -1806,6 +1812,15 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
table_close(heapRel, heapLockmode);
}
+/*
+ * Used to keep track of compressed TID lists when building a GIN tuple.
+ */
+typedef struct
+{
+ dlist_node node; /* linked list pointers */
+ GinPostingList *seg;
+} GinSegmentInfo;
+
/*
* _gin_build_tuple
* Serialize the state for an index key into a tuple for tuplesort.
@@ -1818,6 +1833,11 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
* like endianess etc. We could make it a little bit smaller, but it's not
* worth it - it's a tiny fraction of the data, and we need to MAXALIGN the
* start of the TID list anyway. So we wouldn't save anything.
+ *
+ * The TID list is serialized as compressed - it's highly compressible, and
+ * we already have ginCompressPostingList for this purpose. The list may be
+ * pretty long, so we compress it into multiple segments and then copy all
+ * of that into the GIN tuple.
*/
static GinTuple *
_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
@@ -1831,6 +1851,11 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
Size tuplen;
int keylen;
+ dlist_mutable_iter iter;
+ dlist_head segments;
+ int ncompressed;
+ Size compresslen;
+
/*
* Calculate how long is the key value. Only keys with GIN_CAT_NORM_KEY
* have actual non-empty key. We include varlena headers and \0 bytes for
@@ -1854,12 +1879,34 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
else
elog(ERROR, "invalid typlen");
+ /* compress the item pointers */
+ ncompressed = 0;
+ compresslen = 0;
+ dlist_init(&segments);
+
+ /* generate compressed segments of TID list chunks */
+ while (ncompressed < nitems)
+ {
+ int cnt;
+ GinSegmentInfo *seginfo = palloc(sizeof(GinSegmentInfo));
+
+ seginfo->seg = ginCompressPostingList(&items[ncompressed],
+ (nitems - ncompressed),
+ UINT16_MAX,
+ &cnt);
+
+ ncompressed += cnt;
+ compresslen += SizeOfGinPostingList(seginfo->seg);
+
+ dlist_push_tail(&segments, &seginfo->node);
+ }
+
/*
* Determine GIN tuple length with all the data included. Be careful about
- * alignment, to allow direct access to item pointers.
+ * alignment, to allow direct access to compressed segments (those require
+ * SHORTALIGN, but we do MAXALING anyway).
*/
- tuplen = MAXALIGN(offsetof(GinTuple, data) + keylen) +
- (sizeof(ItemPointerData) * nitems);
+ tuplen = MAXALIGN(offsetof(GinTuple, data) + keylen) + compresslen;
*len = tuplen;
@@ -1909,37 +1956,40 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
/* finally, copy the TIDs into the array */
ptr = (char *) tuple + MAXALIGN(offsetof(GinTuple, data) + keylen);
- memcpy(ptr, items, sizeof(ItemPointerData) * nitems);
+ /* copy in the compressed data, and free the segments */
+ dlist_foreach_modify(iter, &segments)
+ {
+ GinSegmentInfo *seginfo = dlist_container(GinSegmentInfo, node, iter.cur);
+
+ memcpy(ptr, seginfo->seg, SizeOfGinPostingList(seginfo->seg));
+
+ ptr += SizeOfGinPostingList(seginfo->seg);
+
+ dlist_delete(&seginfo->node);
+
+ pfree(seginfo->seg);
+ pfree(seginfo);
+ }
return tuple;
}
/*
- * _gin_parse_tuple
- * Deserialize the tuple from the tuplestore representation.
+ * _gin_parse_tuple_key
+ * Return a Datum representing the key stored in the tuple.
*
- * Most of the fields are actually directly accessible, the only thing that
+ * Most of the tuple fields are directly accessible, the only thing that
* needs more care is the key and the TID list.
*
* For the key, this returns a regular Datum representing it. It's either the
* actual key value, or a pointer to the beginning of the data array (which is
* where the data was copied by _gin_build_tuple).
- *
- * The pointer to the TID list is returned through 'items' (which is simply
- * a pointer to the data array).
*/
static Datum
-_gin_parse_tuple(GinTuple *a, ItemPointerData **items)
+_gin_parse_tuple_key(GinTuple *a)
{
Datum key;
- if (items)
- {
- char *ptr = (char *) a + MAXALIGN(offsetof(GinTuple, data) + a->keylen);
-
- *items = (ItemPointerData *) ptr;
- }
-
if (a->category != GIN_CAT_NORM_KEY)
return (Datum) 0;
@@ -1952,6 +2002,28 @@ _gin_parse_tuple(GinTuple *a, ItemPointerData **items)
return PointerGetDatum(a->data);
}
+/*
+* _gin_parse_tuple_items
+ * Return a pointer to a palloc'd array of decompressed TID array.
+ */
+static ItemPointer
+_gin_parse_tuple_items(GinTuple *a)
+{
+ int len;
+ char *ptr;
+ int ndecoded;
+ ItemPointer items;
+
+ len = a->tuplen - MAXALIGN(offsetof(GinTuple, data) + a->keylen);
+ ptr = (char *) a + MAXALIGN(offsetof(GinTuple, data) + a->keylen);
+
+ items = ginPostingListDecodeAllSegments((GinPostingList *) ptr, len, &ndecoded);
+
+ Assert(ndecoded == a->nitems);
+
+ return (ItemPointer) items;
+}
+
/*
* _gin_compare_tuples
* Compare GIN tuples, used by tuplesort during parallel index build.
@@ -1992,8 +2064,8 @@ _gin_compare_tuples(GinTuple *a, GinTuple *b)
if ((a->category == GIN_CAT_NORM_KEY) &&
(b->category == GIN_CAT_NORM_KEY))
{
- keya = _gin_parse_tuple(a, NULL);
- keyb = _gin_parse_tuple(b, NULL);
+ keya = _gin_parse_tuple_key(a);
+ keyb = _gin_parse_tuple_key(b);
/*
* works for both byval and byref types with fixed lenght, because for
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 0ee1b9ab19b..97b8770ed5f 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1023,6 +1023,7 @@ GinScanEntry
GinScanKey
GinScanOpaque
GinScanOpaqueData
+GinSegmentInfo
GinShared
GinState
GinStatsData
--
2.44.0
[text/x-patch] v20240502-0005-Collect-and-print-compression-stats.patch (5.1K, ../../[email protected]/8-v20240502-0005-Collect-and-print-compression-stats.patch)
download | inline diff:
From 6d726c68fef8969b8c6acba0f6647dd1db83174e Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:43 +0200
Subject: [PATCH v20240502 5/8] Collect and print compression stats
Allows evaluating the benefits of compressing the TID lists.
---
src/backend/access/gin/gininsert.c | 36 +++++++++++++++++++++++++-----
src/include/access/gin.h | 2 ++
2 files changed, 32 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 607ce9b34d6..acfc4e56838 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -190,7 +190,8 @@ static void _gin_parallel_scan_and_build(GinBuildState *buildstate,
static ItemPointer _gin_parse_tuple_items(GinTuple *a);
static Datum _gin_parse_tuple_key(GinTuple *a);
-static GinTuple *_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+static GinTuple *_gin_build_tuple(GinBuildState *state,
+ OffsetNumber attrnum, unsigned char category,
Datum key, int16 typlen, bool typbyval,
ItemPointerData *items, uint32 nitems,
Size *len);
@@ -539,7 +540,7 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
/* there could be many entries, so be willing to abort here */
CHECK_FOR_INTERRUPTS();
- tup = _gin_build_tuple(attnum, category,
+ tup = _gin_build_tuple(buildstate, attnum, category,
key, attr->attlen, attr->attbyval,
list, nlist, &tuplen);
@@ -1530,6 +1531,15 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
/* sort the raw per-worker data */
tuplesort_performsort(state->bs_worker_sort);
+ /* print some basic info */
+ elog(LOG, "_gin_parallel_scan_and_build raw %lu compressed %lu ratio %.2f%%",
+ state->buildStats.sizeRaw, state->buildStats.sizeCompressed,
+ (100.0 * state->buildStats.sizeCompressed) / state->buildStats.sizeRaw);
+
+ /* reset before the second phase */
+ state->buildStats.sizeCompressed = 0;
+ state->buildStats.sizeRaw = 0;
+
/*
* Read the GIN tuples from the shared tuplesort, sorted by the key, and
* merge them into larger chunks for the leader to combine.
@@ -1556,7 +1566,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
*/
AssertCheckItemPointers(buffer, true);
- ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ ntup = _gin_build_tuple(state, buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
buffer->items, buffer->nitems, &ntuplen);
@@ -1583,7 +1593,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
AssertCheckItemPointers(buffer, true);
- ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ ntup = _gin_build_tuple(state, buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
buffer->items, buffer->nitems, &ntuplen);
@@ -1598,6 +1608,11 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
/* relase all the memory */
GinBufferFree(buffer);
+ /* print some basic info */
+ elog(LOG, "_gin_process_worker_data raw %lu compressed %lu ratio %.2f%%",
+ state->buildStats.sizeRaw, state->buildStats.sizeCompressed,
+ (100.0 * state->buildStats.sizeCompressed) / state->buildStats.sizeRaw);
+
tuplesort_end(worker_sort);
}
@@ -1669,7 +1684,7 @@ _gin_parallel_scan_and_build(GinBuildState *state,
/* there could be many entries, so be willing to abort here */
CHECK_FOR_INTERRUPTS();
- tup = _gin_build_tuple(attnum, category,
+ tup = _gin_build_tuple(state, attnum, category,
key, attr->attlen, attr->attbyval,
list, nlist, &len);
@@ -1763,6 +1778,7 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
/* initialize the GIN build state */
initGinState(&buildstate.ginstate, indexRel);
buildstate.indtuples = 0;
+ /* XXX shouldn't this initialize the other fiedls, like ginbuild()? */
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
/*
@@ -1840,7 +1856,8 @@ typedef struct
* of that into the GIN tuple.
*/
static GinTuple *
-_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+_gin_build_tuple(GinBuildState *state,
+ OffsetNumber attrnum, unsigned char category,
Datum key, int16 typlen, bool typbyval,
ItemPointerData *items, uint32 nitems,
Size *len)
@@ -1971,6 +1988,13 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
pfree(seginfo);
}
+ /* how large would the tuple be without compression? */
+ state->buildStats.sizeRaw += MAXALIGN(offsetof(GinTuple, data) + keylen) +
+ nitems * sizeof(ItemPointerData);
+
+ /* compressed size */
+ state->buildStats.sizeCompressed += tuplen;
+
return tuple;
}
diff --git a/src/include/access/gin.h b/src/include/access/gin.h
index be76d8446f4..2b6633d068a 100644
--- a/src/include/access/gin.h
+++ b/src/include/access/gin.h
@@ -49,6 +49,8 @@ typedef struct GinStatsData
BlockNumber nDataPages;
int64 nEntries;
int32 ginVersion;
+ Size sizeRaw;
+ Size sizeCompressed;
} GinStatsData;
/*
--
2.44.0
[text/x-patch] v20240502-0006-Enforce-memory-limit-when-combining-tuples.patch (14.8K, ../../[email protected]/9-v20240502-0006-Enforce-memory-limit-when-combining-tuples.patch)
download | inline diff:
From ace40a448684ffb1fac1e4630b5657d8ffd3d27d Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:49 +0200
Subject: [PATCH v20240502 6/8] Enforce memory limit when combining tuples
When combinnig intermediate results during parallel GIN index build, we
want to restrict the memory usage. In ginBuildCallbackParallel() this is
done simply by dumping working state into tuplesort after hitting the
memory limit.
This commit introduces memory limit to the following steps, merging the
intermediate results in both worker and leader. The merge only deals
with one key at a time, and the primary risk is the key might have too
many different TIDs. This is not very likely, because the TID array only
needs 6B per item, it's a potential issue.
We can't simply dump the whole current TID list - the index requires the
TID values to be inserted in the correct order, but if the lists overlap
(as they do between workers), the tail of the list may change during the
mergesort. But thanks to sorting GIN tuples by first TID, we can derive
a safe TID horizon - we know no future tuples will have TIDs from before
this value, so it's safe to output this part of the list.
This commit tracks "frozen" part of the the TID list, which is the part
we know won't change after merging additional TID lists. Then if the TID
list grows too large (more than 64kB), we try to trim it - write out the
frozen part of the list, and discard it from the buffer. We only do the
trimming if there's at least 1024 frozen items - we don't want to write
the data into the index in tiny chunks.
The freezing also allows us to skip the frozen part during mergesort.
The frozen part of the list is known to be fully sorted, so we can just
skip it and mergesort only the rest of the data.
Note: These limites (1024 and 64kB) are mostly arbitrary - but seem high
enough to get good efficiency for compression/batching, but low enough
to release memory early and work in small increments.
---
src/backend/access/gin/gininsert.c | 245 +++++++++++++++++++++++++++--
src/include/access/gin.h | 1 +
2 files changed, 237 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index acfc4e56838..1d9557692a3 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -1130,8 +1130,12 @@ typedef struct GinBuffer
int16 typlen;
bool typbyval;
+ /* Number of TIDs to collect before attempt to write some out. */
+ int maxitems;
+
/* array of TID values */
int nitems;
+ int nfrozen;
ItemPointerData *items;
} GinBuffer;
@@ -1166,7 +1170,21 @@ AssertCheckItemPointers(GinBuffer *buffer, bool sorted)
static GinBuffer *
GinBufferInit(void)
{
- return palloc0(sizeof(GinBuffer));
+ GinBuffer *buffer = (GinBuffer *) palloc0(sizeof(GinBuffer));
+
+ /*
+ * How many items can we fit into the memory limit? 64kB seems more than
+ * enough and we don't want a limit that's too high. OTOH maybe this
+ * should be tied to maintenance_work_mem or something like that?
+ *
+ * XXX This is not enough to prevent repeated merges after a wraparound,
+ * but it should be enough to make the merges cheap because it quickly
+ * finds reaches the end of the second list and can just memcpy the rest
+ * without walking it item by item.
+ */
+ buffer->maxitems = (64 * 1024L) / sizeof(ItemPointerData);
+
+ return buffer;
}
static bool
@@ -1221,6 +1239,54 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
return (memcmp(tup->data, DatumGetPointer(buffer->key), buffer->keylen) == 0);
}
+/*
+ * GinBufferShouldTrim
+ * Should we trim the list of item pointers?
+ *
+ * By trimming we understand writing out and removing the tuple IDs that
+ * we know can't change by future merges. We can deduce the TID up to which
+ * this is guaranteed from the "first" TID in each GIN tuple, which provides
+ * a "horizon" (for a given key) thanks to the sort.
+ *
+ * We don't want to do this too often - compressing longer TID lists is more
+ * efficient. But we also don't want to accumulate too many TIDs, for two
+ * reasons. First, it consumes memory and we might exceed maintenance_work_mem
+ * (or whatever limit applies), even if that's unlikely because TIDs are very
+ * small so we can fit a lot of them. Second, and more importantly, long TID
+ * lists are an issue if the scan wraps around, because a key may get a very
+ * wide list (with min/max TID for that key), forcing "full" mergesorts for
+ * every list merged into it (instead of the efficient append).
+ *
+ * So we look at two things when deciding if to trim - if the resulting list
+ * (after adding TIDs from the new tuple) would be too long, and if there is
+ * enough TIDs to trim (with values less than "first" TID from the new tuple),
+ * we do the trim. By enough we mean at least 128 TIDs (mostly an arbitrary
+ * number).
+ *
+ * XXX This does help for the wrap around case, because the "wide" TID list
+ * is essentially two ranges - one at the beginning of the table, one at the
+ * end. And all the other ranges (from GIN tuples) come in between, and also
+ * do not overlap. So by trimming up to the range we're about to add, this
+ * guarantees we'll be able to "concatenate" the two lists cheaply.
+ */
+static bool
+GinBufferShouldTrim(GinBuffer *buffer, GinTuple *tup)
+{
+ /* not enough TIDs to trim (1024 is somewhat arbitrary number) */
+ if (buffer->nfrozen < 1024)
+ return false;
+
+ /* We're not to hit the memory limit after adding this tuple. */
+ if ((buffer->nitems + tup->nitems) < buffer->maxitems)
+ return false;
+
+ /*
+ * OK, we have enough frozen TIDs to flush, and we have hit the memory
+ * limit, so it's time to write it out.
+ */
+ return true;
+}
+
/*
* GinBufferStoreTuple
* Add data from a GinTuple into the GinBuffer.
@@ -1259,6 +1325,11 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
* once, so there's going to be only such wide list, and it'll be sorted
* first (because it has the lowest TID for the key). So we'd do this at
* most once per key.
+ *
+ * XXX Maybe we could/should allocate the buffer once and then keep it
+ * without palloc/pfree. That won't help when just calling the mergesort,
+ * as that does palloc internally, but if we detected the append case,
+ * we could do without it. Not sure how much overhead it is, though.
*/
static void
GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
@@ -1287,26 +1358,82 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->key = (Datum) 0;
}
+ /*
+ * Try freeze TIDs at the beginning of the list, i.e. exclude them from
+ * the mergesort. We can do that with TIDs before the first TID in the new
+ * tuple we're about to add into the buffer.
+ *
+ * We do this incrementally when adding data into the in-memory buffer,
+ * and not later (e.g. when hitting a memory limit), because it allows us
+ * to skip the frozen data during the mergesort, making it cheaper.
+ */
+
+ /*
+ * Check if the last TID in the current list is frozen. This is the case
+ * when merging non-overlapping lists, e.g. in each parallel worker.
+ */
+ if ((buffer->nitems > 0) &&
+ (ItemPointerCompare(&buffer->items[buffer->nitems - 1], &tup->first) == 0))
+ buffer->nfrozen = buffer->nitems;
+
+ /*
+ * Now search the list linearly, to find the last frozen TID. If we found
+ * the whole list is frozen, this just does nothing.
+ *
+ * Start with the first not-yet-frozen tuple, and walk until we find the
+ * first TID that's higher.
+ *
+ * XXX Maybe this should do a binary search if the number of "non-frozen"
+ * items is sufficiently high (enough to make linear search slower than
+ * binsearch).
+ */
+ for (int i = buffer->nfrozen; i < buffer->nitems; i++)
+ {
+ /* Is the TID after the first TID of the new tuple? Can't freeze. */
+ if (ItemPointerCompare(&buffer->items[i], &tup->first) > 0)
+ break;
+
+ buffer->nfrozen++;
+ }
+
/*
* Copy the new TIDs into the buffer, combine with existing data (if any)
* using merge-sort. The mergesort is already smart about cases where it
* can simply concatenate the two lists, and when it actually needs to
* merge the data in an expensive way.
+ *
+ * XXX We could check if (buffer->nitems > buffer->nfrozen) and only do
+ * the mergesort in that case. ginMergeItemPointers does some palloc
+ * internally, and this way we could eliminate that. But let's keep the
+ * code simple for now.
*/
{
int nnew;
ItemPointer new;
- new = ginMergeItemPointers(buffer->items, buffer->nitems,
+ /*
+ * Resize the array - we do this first, because we'll dereference the
+ * first unfrozen TID, which would fail if the array is NULL. We'll
+ * still pass 0 as number of elements in that array though.
+ */
+ if (buffer->items == NULL)
+ buffer->items = palloc((buffer->nitems + tup->nitems) * sizeof(ItemPointerData));
+ else
+ buffer->items = repalloc(buffer->items,
+ (buffer->nitems + tup->nitems) * sizeof(ItemPointerData));
+
+ new = ginMergeItemPointers(&buffer->items[buffer->nfrozen], /* first unfronzen */
+ (buffer->nitems - buffer->nfrozen), /* num of unfrozen */
items, tup->nitems, &nnew);
- Assert(nnew == buffer->nitems + tup->nitems);
+ Assert(nnew == (tup->nitems + (buffer->nitems - buffer->nfrozen)));
+
+ memcpy(&buffer->items[buffer->nfrozen], new,
+ nnew * sizeof(ItemPointerData));
- if (buffer->items)
- pfree(buffer->items);
+ pfree(new);
- buffer->items = new;
- buffer->nitems = nnew;
+ buffer->nitems += tup->nitems;
AssertCheckItemPointers(buffer, true);
}
@@ -1332,6 +1459,7 @@ GinBufferReset(GinBuffer *buffer)
buffer->category = 0;
buffer->keylen = 0;
buffer->nitems = 0;
+ buffer->nfrozen = 0;
buffer->typlen = 0;
buffer->typbyval = 0;
@@ -1344,6 +1472,23 @@ GinBufferReset(GinBuffer *buffer)
/* XXX should do something with extremely large array of items? */
}
+/*
+ * GinBufferTrim
+ * Discard the "frozen" part of the TID list (which should have been
+ * written to disk/index before this call).
+ */
+static void
+GinBufferTrim(GinBuffer *buffer)
+{
+ Assert((buffer->nfrozen > 0) && (buffer->nfrozen <= buffer->nitems));
+
+ memmove(&buffer->items[0], &buffer->items[buffer->nfrozen],
+ sizeof(ItemPointerData) * (buffer->nitems - buffer->nfrozen));
+
+ buffer->nitems -= buffer->nfrozen;
+ buffer->nfrozen = 0;
+}
+
/* XXX probably would be better to have a memory context for the buffer */
static void
GinBufferFree(GinBuffer *buffer)
@@ -1402,7 +1547,12 @@ _gin_parallel_merge(GinBuildState *state)
/* do the actual sort in the leader */
tuplesort_performsort(state->bs_sortstate);
- /* initialize buffer to combine entries for the same key */
+ /*
+ * Initialize buffer to combine entries for the same key.
+ *
+ * The leader is allowed to use the whole maintenance_work_mem buffer to
+ * combine data. The parallel workers already completed.
+ */
buffer = GinBufferInit();
/*
@@ -1442,6 +1592,36 @@ _gin_parallel_merge(GinBuildState *state)
GinBufferReset(buffer);
}
+ /*
+ * We're about to add a GIN tuple to the buffer - check the memory
+ * limit first, and maybe write out some of the data into the index
+ * first, if needed (and possible). We only flush the part of the TID
+ * list that we know won't change, and only if there's enough data for
+ * compression to work well.
+ *
+ * XXX The buffer may also be empty, but in that case we skip this.
+ */
+ if (GinBufferShouldTrim(buffer, tup))
+ {
+ Assert(buffer->nfrozen > 0);
+
+ state->buildStats.nTrims++;
+
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ AssertCheckItemPointers(buffer, true);
+
+ ginEntryInsert(&state->ginstate,
+ buffer->attnum, buffer->key, buffer->category,
+ buffer->items, buffer->nfrozen, &state->buildStats);
+
+ /* truncate the data we've just discarded */
+ GinBufferTrim(buffer);
+ }
+
/*
* Remember data for the current tuple (either remember the new key,
* or append if to the existing data).
@@ -1465,6 +1645,8 @@ _gin_parallel_merge(GinBuildState *state)
/* relase all the memory */
GinBufferFree(buffer);
+ elog(LOG, "_gin_parallel_merge ntrims %ld", state->buildStats.nTrims);
+
tuplesort_end(state->bs_sortstate);
return reltuples;
@@ -1525,7 +1707,13 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinBuffer *buffer;
- /* initialize buffer to combine entries for the same key */
+ /*
+ * Initialize buffer to combine entries for the same key.
+ *
+ * The workers are limited to the same amount of memory as during the sort
+ * in ginBuildCallbackParallel. But this probably should be the 32MB used
+ * during planning, just like there.
+ */
buffer = GinBufferInit();
/* sort the raw per-worker data */
@@ -1578,6 +1766,43 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinBufferReset(buffer);
}
+ /*
+ * We're about to add a GIN tuple to the buffer - check the memory
+ * limit first, and maybe write out some of the data into the index
+ * first, if needed (and possible). We only flush the part of the TID
+ * list that we know won't change, and only if there's enough data for
+ * compression to work well.
+ *
+ * XXX The buffer may also be empty, but in that case we skip this.
+ */
+ if (GinBufferShouldTrim(buffer, tup))
+ {
+ GinTuple *ntup;
+ Size ntuplen;
+
+ Assert(buffer->nfrozen > 0);
+
+ state->buildStats.nTrims++;
+
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ AssertCheckItemPointers(buffer, true);
+
+ ntup = _gin_build_tuple(state, buffer->attnum, buffer->category,
+ buffer->key, buffer->typlen, buffer->typbyval,
+ buffer->items, buffer->nfrozen, &ntuplen);
+
+ tuplesort_putgintuple(state->bs_sortstate, ntup, ntuplen);
+
+ pfree(ntup);
+
+ /* truncate the data we've just discarded */
+ GinBufferTrim(buffer);
+ }
+
/*
* Remember data for the current tuple (either remember the new key,
* or append if to the existing data).
@@ -1613,6 +1838,8 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
state->buildStats.sizeRaw, state->buildStats.sizeCompressed,
(100.0 * state->buildStats.sizeCompressed) / state->buildStats.sizeRaw);
+ elog(LOG, "_gin_process_worker_data trims %ld", state->buildStats.nTrims);
+
tuplesort_end(worker_sort);
}
diff --git a/src/include/access/gin.h b/src/include/access/gin.h
index 2b6633d068a..9381329fac5 100644
--- a/src/include/access/gin.h
+++ b/src/include/access/gin.h
@@ -51,6 +51,7 @@ typedef struct GinStatsData
int32 ginVersion;
Size sizeRaw;
Size sizeCompressed;
+ int64 nTrims;
} GinStatsData;
/*
--
2.44.0
[text/x-patch] v20240502-0007-Detect-wrap-around-in-parallel-callback.patch (5.7K, ../../[email protected]/10-v20240502-0007-Detect-wrap-around-in-parallel-callback.patch)
download | inline diff:
From a569086cd2dc70ee0b9152980eb2301d99f8c580 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:55 +0200
Subject: [PATCH v20240502 7/8] Detect wrap around in parallel callback
When sync scan during index build wraps around, that may result in some
keys having very long TID lists, requiring "full" merge sort runs when
combining data in workers. It also causes problems with enforcing memory
limit, because we can't just dump the data - the index build requires
append-only posting lists, and violating may result in errors like
ERROR: could not split GIN page; all old items didn't fit
because after the scan wrap around some of the TIDs may belong to the
beginning of the list, affecting the compression.
But we can deal with this in the callback - if we see the TID to jump
back, that must mean a wraparound happened. In that case we simply dump
all the data accumulated in memory, and start from scratch.
This means there won't be any tuples with very wide TID ranges, instead
there'll be one tuple with a range at the end of the table, and another
tuple at the beginning. And all the lists in the worker will be
non-overlapping, and sort nicely based on first TID.
For the leader, we still need to do the full merge - the lists may
overlap and interleave in various ways. But there should be only very
few of those lists, about one per worker, making it not an issue.
---
src/backend/access/gin/gininsert.c | 89 ++++++++++++++++++------------
1 file changed, 55 insertions(+), 34 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 1d9557692a3..acdf45416fe 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -142,6 +142,7 @@ typedef struct
MemoryContext tmpCtx;
MemoryContext funcCtx;
BuildAccumulator accum;
+ ItemPointerData tid;
/* FIXME likely duplicate with indtuples */
double bs_numtuples;
@@ -474,6 +475,47 @@ ginBuildCallback(Relation index, ItemPointer tid, Datum *values,
MemoryContextSwitchTo(oldCtx);
}
+static void
+ginFlushBuildState(GinBuildState *buildstate, Relation index)
+{
+ ItemPointerData *list;
+ Datum key;
+ GinNullCategory category;
+ uint32 nlist;
+ OffsetNumber attnum;
+ TupleDesc tdesc = RelationGetDescr(index);
+
+ ginBeginBAScan(&buildstate->accum);
+ while ((list = ginGetBAEntry(&buildstate->accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* information about the key */
+ Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
+
+ /* GIN tuple and tuple length */
+ GinTuple *tup;
+ Size tuplen;
+
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+
+ tup = _gin_build_tuple(buildstate, attnum, category,
+ key, attr->attlen, attr->attbyval,
+ list, nlist, &tuplen);
+
+ tuplesort_putgintuple(buildstate->bs_worker_sort, tup, tuplen);
+
+ pfree(tup);
+ }
+
+ MemoryContextReset(buildstate->tmpCtx);
+ ginInitBA(&buildstate->accum);
+}
+
+/*
+ * FIXME Another way to deal with the wrap around of sync scans would be to
+ * detect when tid wraps around and just flush the state.
+ */
static void
ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
bool *isnull, bool tupleIsAlive, void *state)
@@ -484,6 +526,16 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
oldCtx = MemoryContextSwitchTo(buildstate->tmpCtx);
+ /* flush contents before wrapping around */
+ if (ItemPointerCompare(tid, &buildstate->tid) < 0)
+ {
+ elog(LOG, "calling ginFlushBuildState");
+ ginFlushBuildState(buildstate, index);
+ }
+
+ /* remember the TID we're about to process */
+ memcpy(&buildstate->tid, tid, sizeof(ItemPointerData));
+
for (i = 0; i < buildstate->ginstate.origTupdesc->natts; i++)
ginHeapTupleBulkInsert(buildstate, (OffsetNumber) (i + 1),
values[i], isnull[i], tid);
@@ -518,40 +570,7 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
* XXX probably should use 32MB, not work_mem, as used during planning?
*/
if (buildstate->accum.allocatedMemory >= (Size) work_mem * 1024L)
- {
- ItemPointerData *list;
- Datum key;
- GinNullCategory category;
- uint32 nlist;
- OffsetNumber attnum;
- TupleDesc tdesc = RelationGetDescr(index);
-
- ginBeginBAScan(&buildstate->accum);
- while ((list = ginGetBAEntry(&buildstate->accum,
- &attnum, &key, &category, &nlist)) != NULL)
- {
- /* information about the key */
- Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
-
- /* GIN tuple and tuple length */
- GinTuple *tup;
- Size tuplen;
-
- /* there could be many entries, so be willing to abort here */
- CHECK_FOR_INTERRUPTS();
-
- tup = _gin_build_tuple(buildstate, attnum, category,
- key, attr->attlen, attr->attbyval,
- list, nlist, &tuplen);
-
- tuplesort_putgintuple(buildstate->bs_worker_sort, tup, tuplen);
-
- pfree(tup);
- }
-
- MemoryContextReset(buildstate->tmpCtx);
- ginInitBA(&buildstate->accum);
- }
+ ginFlushBuildState(buildstate, index);
MemoryContextSwitchTo(oldCtx);
}
@@ -587,6 +606,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.bs_numtuples = 0;
buildstate.bs_reltuples = 0;
buildstate.bs_leader = NULL;
+ memset(&buildstate.tid, 0, sizeof(ItemPointerData));
/* initialize the meta page */
MetaBuffer = GinNewBuffer(index);
@@ -2007,6 +2027,7 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
buildstate.indtuples = 0;
/* XXX shouldn't this initialize the other fiedls, like ginbuild()? */
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
+ memset(&buildstate.tid, 0, sizeof(ItemPointerData));
/*
* create a temporary memory context that is used to hold data not yet
--
2.44.0
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Parallel CREATE INDEX for GIN indexes
@ 2024-05-02 17:12 Matthias van de Meent <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 1 reply; 7+ messages in thread
From: Matthias van de Meent @ 2024-05-02 17:12 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On Thu, 2 May 2024 at 17:19, Tomas Vondra <[email protected]> wrote:
>
> Hi,
>
> In PG17 we shall have parallel CREATE INDEX for BRIN indexes, and back
> when working on that I was thinking how difficult would it be to do
> something similar to do that for other index types, like GIN. I even had
> that on my list of ideas to pitch to potential contributors, as I was
> fairly sure it's doable and reasonably isolated / well-defined.
>
> However, I was not aware of any takers, so a couple days ago on a slow
> weekend I took a stab at it. And yes, it's doable - attached is a fairly
> complete, tested and polished version of the feature, I think. It turned
> out to be a bit more complex than I expected, for reasons that I'll get
> into when discussing the patches.
This is great. I've been thinking about approximately the same issue
recently, too, but haven't had time to discuss/implement any of this
yet. I think some solutions may even be portable to the btree parallel
build: it also has key deduplication (though to a much smaller degree)
and could benefit from deduplication during the scan/ssup load phase,
rather than only during insertion.
> First, let's talk about the benefits - how much faster is that than the
> single-process build we have for GIN indexes? I do have a table with the
> archive of all our mailing lists - it's ~1.5M messages, table is ~21GB
> (raw dump is about 28GB). This does include simple text data (message
> body), JSONB (headers) and tsvector (full-text on message body).
Sidenote: Did you include the tsvector in the table to reduce time
spent during index creation? I would have used an expression in the
index definition, rather than a direct column.
> If I do CREATE index with different number of workers (0 means serial
> build), I get this timings (in seconds):
[...]
> This shows the benefits are pretty nice, depending on the opclass. For
> most indexes it's maybe ~3-4x faster, which is nice, and I don't think
> it's possible to do much better - the actual index inserts can happen
> from a single process only, which is the main limit.
Can we really not insert with multiple processes? It seems to me that
GIN could be very suitable for that purpose, with its clear double
tree structure distinction that should result in few buffer conflicts
if different backends work on known-to-be-very-different keys.
We'd probably need multiple read heads on the shared tuplesort, and a
way to join the generated top-level subtrees, but I don't think that
is impossible. Maybe it's work for later effort though.
Have you tested and/or benchmarked this with multi-column GIN indexes?
> For some of the opclasses it can regress (like the jsonb_path_ops). I
> don't think that's a major issue. Or more precisely, I'm not surprised
> by it. It'd be nice to be able to disable the parallel builds in these
> cases somehow, but I haven't thought about that.
Do you know why it regresses?
> I do plan to do some tests with btree_gin, but I don't expect that to
> behave significantly differently.
>
> There are small variations in the index size, when built in the serial
> way and the parallel way. It's generally within ~5-10%, and I believe
> it's due to the serial build adding the TIDs incrementally, while the
> build adds them in much larger chunks (possibly even in one chunk with
> all the TIDs for the key).
I assume that was '[...] while the [parallel] build adds them [...]', right?
> I believe the same size variation can happen
> if the index gets built in a different way, e.g. by inserting the data
> in a different order, etc. I did a number of tests to check if the index
> produces the correct results, and I haven't found any issues. So I think
> this is OK, and neither a problem nor an advantage of the patch.
>
>
> Now, let's talk about the code - the series has 7 patches, with 6
> non-trivial parts doing changes in focused and easier to understand
> pieces (I hope so).
The following comments are generally based on the descriptions; I
haven't really looked much at the patches yet except to validate some
assumptions.
> 1) v20240502-0001-Allow-parallel-create-for-GIN-indexes.patch
>
> This is the initial feature, adding the "basic" version, implemented as
> pretty much 1:1 copy of the BRIN parallel build and minimal changes to
> make it work for GIN (mostly about how to store intermediate results).
>
> The basic idea is that the workers do the regular build, but instead of
> flushing the data into the index after hitting the memory limit, it gets
> written into a shared tuplesort and sorted by the index key. And the
> leader then reads this sorted data, accumulates the TID for a given key
> and inserts that into the index in one go.
In the code, GIN insertions are still basically single btree
insertions, all starting from the top (but with many same-valued
tuples at once). Now that we have a tuplesort with the full table's
data, couldn't the code be adapted to do more efficient btree loading,
such as that seen in the nbtree code, where the rightmost pages are
cached and filled sequentially without requiring repeated searches
down the tree? I suspect we can gain a lot of time there.
I don't need you to do that, but what's your opinion on this?
> 2) v20240502-0002-Use-mergesort-in-the-leader-process.patch
>
> The approach implemented by 0001 works, but there's a little bit of
> issue - if there are many distinct keys (e.g. for trigrams that can
> happen very easily), the workers will hit the memory limit with only
> very short TID lists for most keys. For serial build that means merging
> the data into a lot of random places, and in parallel build it means the
> leader will have to merge a lot of tiny lists from many sorted rows.
>
> Which can be quite annoying and expensive, because the leader does so
> using qsort() in the serial part. It'd be better to ensure most of the
> sorting happens in the workers, and the leader can do a mergesort. But
> the mergesort must not happen too often - merging many small lists is
> not cheaper than a single qsort (especially when the lists overlap).
>
> So this patch changes the workers to process the data in two phases. The
> first works as before, but the data is flushed into a local tuplesort.
> And then each workers sorts the results it produced, and combines them
> into results with much larger TID lists, and those results are written
> to the shared tuplesort. So the leader only gets very few lists to
> combine for a given key - usually just one list per worker.
Hmm, I was hoping we could implement the merging inside the tuplesort
itself during its own flush phase, as it could save significantly on
IO, and could help other users of tuplesort with deduplication, too.
> 3) v20240502-0003-Remove-the-explicit-pg_qsort-in-workers.patch
>
> In 0002 the workers still do an explicit qsort() on the TID list before
> writing the data into the shared tuplesort. But we can do better - the
> workers can do a merge sort too. To help with this, we add the first TID
> to the tuplesort tuple, and sort by that too - it helps the workers to
> process the data in an order that allows simple concatenation instead of
> the full mergesort.
>
> Note: There's a non-obvious issue due to parallel scans always being
> "sync scans", which may lead to very "wide" TID ranges when the scan
> wraps around. More about that later.
As this note seems to imply, this seems to have a strong assumption
that data received in parallel workers is always in TID order, with
one optional wraparound. Non-HEAP TAMs may break with this assumption,
so what's the plan on that?
> 4) v20240502-0004-Compress-TID-lists-before-writing-tuples-t.patch
>
> The parallel build passes data between processes using temporary files,
> which means it may need significant amount of disk space. For BRIN this
> was not a major concern, because the summaries tend to be pretty small.
>
> But for GIN that's not the case, and the two-phase processing introduced
> by 0002 make it worse, because the worker essentially creates another
> copy of the intermediate data. It does not need to copy the key, so
> maybe it's not exactly 2x the space requirement, but in the worst case
> it's not far from that.
>
> But there's a simple way how to improve this - the TID lists tend to be
> very compressible, and GIN already implements a very light-weight TID
> compression, so this patch does just that - when building the tuple to
> be written into the tuplesort, we just compress the TIDs.
See note on 0002: Could we do this in the tuplesort writeback, rather
than by moving the data around multiple times?
[...]
> So 0007 does something similar - it tracks if the TID value goes
> backward in the callback, and if it does it dumps the state into the
> tuplesort before processing the first tuple from the beginning of the
> table. Which means we end up with two separate "narrow" TID list, not
> one very wide one.
See note above: We may still need a merge phase, just to make sure we
handle all TAM parallel scans correctly, even if that merge join phase
wouldn't get hit in vanilla PostgreSQL.
Kind regards,
Matthias van de Meent
Neon (https://neon.tech)
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Parallel CREATE INDEX for GIN indexes
@ 2024-05-02 18:22 Tomas Vondra <[email protected]>
parent: Matthias van de Meent <[email protected]>
0 siblings, 2 replies; 7+ messages in thread
From: Tomas Vondra @ 2024-05-02 18:22 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 5/2/24 19:12, Matthias van de Meent wrote:
> On Thu, 2 May 2024 at 17:19, Tomas Vondra <[email protected]> wrote:
>>
>> Hi,
>>
>> In PG17 we shall have parallel CREATE INDEX for BRIN indexes, and back
>> when working on that I was thinking how difficult would it be to do
>> something similar to do that for other index types, like GIN. I even had
>> that on my list of ideas to pitch to potential contributors, as I was
>> fairly sure it's doable and reasonably isolated / well-defined.
>>
>> However, I was not aware of any takers, so a couple days ago on a slow
>> weekend I took a stab at it. And yes, it's doable - attached is a fairly
>> complete, tested and polished version of the feature, I think. It turned
>> out to be a bit more complex than I expected, for reasons that I'll get
>> into when discussing the patches.
>
> This is great. I've been thinking about approximately the same issue
> recently, too, but haven't had time to discuss/implement any of this
> yet. I think some solutions may even be portable to the btree parallel
> build: it also has key deduplication (though to a much smaller degree)
> and could benefit from deduplication during the scan/ssup load phase,
> rather than only during insertion.
>
Perhaps, although I'm not that familiar with the details of btree
builds, and I haven't thought about it when working on this over the
past couple days.
>> First, let's talk about the benefits - how much faster is that than the
>> single-process build we have for GIN indexes? I do have a table with the
>> archive of all our mailing lists - it's ~1.5M messages, table is ~21GB
>> (raw dump is about 28GB). This does include simple text data (message
>> body), JSONB (headers) and tsvector (full-text on message body).
>
> Sidenote: Did you include the tsvector in the table to reduce time
> spent during index creation? I would have used an expression in the
> index definition, rather than a direct column.
>
Yes, it's a materialized column, not computed during index creation.
>> If I do CREATE index with different number of workers (0 means serial
>> build), I get this timings (in seconds):
>
> [...]
>
>> This shows the benefits are pretty nice, depending on the opclass. For
>> most indexes it's maybe ~3-4x faster, which is nice, and I don't think
>> it's possible to do much better - the actual index inserts can happen
>> from a single process only, which is the main limit.
>
> Can we really not insert with multiple processes? It seems to me that
> GIN could be very suitable for that purpose, with its clear double
> tree structure distinction that should result in few buffer conflicts
> if different backends work on known-to-be-very-different keys.
> We'd probably need multiple read heads on the shared tuplesort, and a
> way to join the generated top-level subtrees, but I don't think that
> is impossible. Maybe it's work for later effort though.
>
Maybe, but I took it as a restriction and it seemed too difficult to
relax (or at least I assume that).
> Have you tested and/or benchmarked this with multi-column GIN indexes?
>
I did test that, and I'm not aware of any bugs/issues. Performance-wise
it depends on which opclasses are used by the columns - if you take the
speedup for each of them independently, the speedup for the whole index
is roughly the average of that.
>> For some of the opclasses it can regress (like the jsonb_path_ops). I
>> don't think that's a major issue. Or more precisely, I'm not surprised
>> by it. It'd be nice to be able to disable the parallel builds in these
>> cases somehow, but I haven't thought about that.
>
> Do you know why it regresses?
>
No, but one thing that stands out is that the index is much smaller than
the other columns/opclasses, and the compression does not save much
(only about 5% for both phases). So I assume it's the overhead of
writing writing and reading a bunch of GB of data without really gaining
much from doing that.
>> I do plan to do some tests with btree_gin, but I don't expect that to
>> behave significantly differently.
>>
>> There are small variations in the index size, when built in the serial
>> way and the parallel way. It's generally within ~5-10%, and I believe
>> it's due to the serial build adding the TIDs incrementally, while the
>> build adds them in much larger chunks (possibly even in one chunk with
>> all the TIDs for the key).
>
> I assume that was '[...] while the [parallel] build adds them [...]', right?
>
Right. The parallel build adds them in larger chunks.
>> I believe the same size variation can happen
>> if the index gets built in a different way, e.g. by inserting the data
>> in a different order, etc. I did a number of tests to check if the index
>> produces the correct results, and I haven't found any issues. So I think
>> this is OK, and neither a problem nor an advantage of the patch.
>>
>>
>> Now, let's talk about the code - the series has 7 patches, with 6
>> non-trivial parts doing changes in focused and easier to understand
>> pieces (I hope so).
>
> The following comments are generally based on the descriptions; I
> haven't really looked much at the patches yet except to validate some
> assumptions.
>
OK
>> 1) v20240502-0001-Allow-parallel-create-for-GIN-indexes.patch
>>
>> This is the initial feature, adding the "basic" version, implemented as
>> pretty much 1:1 copy of the BRIN parallel build and minimal changes to
>> make it work for GIN (mostly about how to store intermediate results).
>>
>> The basic idea is that the workers do the regular build, but instead of
>> flushing the data into the index after hitting the memory limit, it gets
>> written into a shared tuplesort and sorted by the index key. And the
>> leader then reads this sorted data, accumulates the TID for a given key
>> and inserts that into the index in one go.
>
> In the code, GIN insertions are still basically single btree
> insertions, all starting from the top (but with many same-valued
> tuples at once). Now that we have a tuplesort with the full table's
> data, couldn't the code be adapted to do more efficient btree loading,
> such as that seen in the nbtree code, where the rightmost pages are
> cached and filled sequentially without requiring repeated searches
> down the tree? I suspect we can gain a lot of time there.
>
> I don't need you to do that, but what's your opinion on this?
>
I have no idea. I started working on this with only very basic idea of
how GIN works / is structured, so I simply leveraged the existing
callback and massaged it to work in the parallel case too.
>> 2) v20240502-0002-Use-mergesort-in-the-leader-process.patch
>>
>> The approach implemented by 0001 works, but there's a little bit of
>> issue - if there are many distinct keys (e.g. for trigrams that can
>> happen very easily), the workers will hit the memory limit with only
>> very short TID lists for most keys. For serial build that means merging
>> the data into a lot of random places, and in parallel build it means the
>> leader will have to merge a lot of tiny lists from many sorted rows.
>>
>> Which can be quite annoying and expensive, because the leader does so
>> using qsort() in the serial part. It'd be better to ensure most of the
>> sorting happens in the workers, and the leader can do a mergesort. But
>> the mergesort must not happen too often - merging many small lists is
>> not cheaper than a single qsort (especially when the lists overlap).
>>
>> So this patch changes the workers to process the data in two phases. The
>> first works as before, but the data is flushed into a local tuplesort.
>> And then each workers sorts the results it produced, and combines them
>> into results with much larger TID lists, and those results are written
>> to the shared tuplesort. So the leader only gets very few lists to
>> combine for a given key - usually just one list per worker.
>
> Hmm, I was hoping we could implement the merging inside the tuplesort
> itself during its own flush phase, as it could save significantly on
> IO, and could help other users of tuplesort with deduplication, too.
>
Would that happen in the worker or leader process? Because my goal was
to do the expensive part in the worker, because that's what helps with
the parallelization.
>> 3) v20240502-0003-Remove-the-explicit-pg_qsort-in-workers.patch
>>
>> In 0002 the workers still do an explicit qsort() on the TID list before
>> writing the data into the shared tuplesort. But we can do better - the
>> workers can do a merge sort too. To help with this, we add the first TID
>> to the tuplesort tuple, and sort by that too - it helps the workers to
>> process the data in an order that allows simple concatenation instead of
>> the full mergesort.
>>
>> Note: There's a non-obvious issue due to parallel scans always being
>> "sync scans", which may lead to very "wide" TID ranges when the scan
>> wraps around. More about that later.
>
> As this note seems to imply, this seems to have a strong assumption
> that data received in parallel workers is always in TID order, with
> one optional wraparound. Non-HEAP TAMs may break with this assumption,
> so what's the plan on that?
>
Well, that would break the serial build too, right? Anyway, the way this
patch works can be extended to deal with that by actually sorting the
TIDs when serializing the tuplestore tuple. The consequence of that is
the combining will be more expensive, because it'll require a proper
mergesort, instead of just appending the lists.
>> 4) v20240502-0004-Compress-TID-lists-before-writing-tuples-t.patch
>>
>> The parallel build passes data between processes using temporary files,
>> which means it may need significant amount of disk space. For BRIN this
>> was not a major concern, because the summaries tend to be pretty small.
>>
>> But for GIN that's not the case, and the two-phase processing introduced
>> by 0002 make it worse, because the worker essentially creates another
>> copy of the intermediate data. It does not need to copy the key, so
>> maybe it's not exactly 2x the space requirement, but in the worst case
>> it's not far from that.
>>
>> But there's a simple way how to improve this - the TID lists tend to be
>> very compressible, and GIN already implements a very light-weight TID
>> compression, so this patch does just that - when building the tuple to
>> be written into the tuplesort, we just compress the TIDs.
>
> See note on 0002: Could we do this in the tuplesort writeback, rather
> than by moving the data around multiple times?
>
No idea, I've never done that ...
> [...]
>> So 0007 does something similar - it tracks if the TID value goes
>> backward in the callback, and if it does it dumps the state into the
>> tuplesort before processing the first tuple from the beginning of the
>> table. Which means we end up with two separate "narrow" TID list, not
>> one very wide one.
>
> See note above: We may still need a merge phase, just to make sure we
> handle all TAM parallel scans correctly, even if that merge join phase
> wouldn't get hit in vanilla PostgreSQL.
>
Well, yeah. But in fact the parallel code already does that, while the
existing serial code may fail with the "data don't fit" error.
The parallel code will do the mergesort correctly, and only emit TIDs
that we know are safe to write to the index (i.e. no future TIDs will go
before the "TID horizon").
But the serial build has nothing like that - it will sort the TIDs that
fit into the memory limit, but it also relies on not processing data out
of order (and disables sync scans to not have wrap around issues). But
if the TAM does something funny, this may break.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Parallel CREATE INDEX for GIN indexes
@ 2024-05-05 18:49 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 7+ messages in thread
From: Tomas Vondra @ 2024-05-05 18:49 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
Hi,
Here's a slightly improved version, fixing a couple bugs in handling
byval/byref values, causing issues on 32-bit machines (but not only).
And also a couple compiler warnings about string formatting.
Other than that, no changes.
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[text/x-patch] v20240505-0001-Allow-parallel-create-for-GIN-indexes.patch (56.4K, ../../[email protected]/2-v20240505-0001-Allow-parallel-create-for-GIN-indexes.patch)
download | inline diff:
From 510af00802c04b8d6d3982069c96082572a76c72 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:26 +0200
Subject: [PATCH v20240505 1/8] Allow parallel create for GIN indexes
Add support for parallel create of GIN indexes, using an approach and
code very similar to the one used by BRIN indexes.
Each worker reads a subset of the table (from a parallel scan), and
accumulated index entries in memory. But instead of writing the results
into the index (after hitting the memory limit), the data are written
to a shared tuplesort (and sorted by index key).
The leader then reads data from the tuplesort, and combines them into
entries that get inserted into the index.
---
src/backend/access/gin/ginbulk.c | 7 +
src/backend/access/gin/gininsert.c | 1340 +++++++++++++++++++-
src/backend/access/gin/ginutil.c | 2 +-
src/backend/access/transam/parallel.c | 4 +
src/backend/utils/sort/tuplesortvariants.c | 154 +++
src/include/access/gin.h | 4 +
src/include/access/gin_tuple.h | 29 +
src/include/utils/tuplesort.h | 6 +
src/tools/pgindent/typedefs.list | 4 +
9 files changed, 1534 insertions(+), 16 deletions(-)
create mode 100644 src/include/access/gin_tuple.h
diff --git a/src/backend/access/gin/ginbulk.c b/src/backend/access/gin/ginbulk.c
index a522801c2f7..12eeff04a6c 100644
--- a/src/backend/access/gin/ginbulk.c
+++ b/src/backend/access/gin/ginbulk.c
@@ -153,6 +153,13 @@ ginInsertBAEntry(BuildAccumulator *accum,
GinEntryAccumulator *ea;
bool isNew;
+ /*
+ * FIXME prevents writes of uninitialized bytes reported by valgrind in
+ * writetup (likely that build_gin_tuple copies some fields that are only
+ * initialized for a certain category, or something similar)
+ */
+ memset(&eatmp, 0, sizeof(GinEntryAccumulator));
+
/*
* For the moment, fill only the fields of eatmp that will be looked at by
* cmpEntryAccumulator or ginCombineData.
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 71f38be90c3..b353e155fc6 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -15,14 +15,124 @@
#include "postgres.h"
#include "access/gin_private.h"
+#include "access/gin_tuple.h"
+#include "access/table.h"
#include "access/tableam.h"
#include "access/xloginsert.h"
+#include "catalog/index.h"
+#include "commands/vacuum.h"
#include "miscadmin.h"
#include "nodes/execnodes.h"
+#include "pgstat.h"
#include "storage/bufmgr.h"
#include "storage/predicate.h"
+#include "tcop/tcopprot.h" /* pgrminclude ignore */
+#include "utils/datum.h"
#include "utils/memutils.h"
#include "utils/rel.h"
+#include "utils/builtins.h"
+
+
+/* Magic numbers for parallel state sharing */
+#define PARALLEL_KEY_GIN_SHARED UINT64CONST(0xB000000000000001)
+#define PARALLEL_KEY_TUPLESORT UINT64CONST(0xB000000000000002)
+#define PARALLEL_KEY_QUERY_TEXT UINT64CONST(0xB000000000000003)
+#define PARALLEL_KEY_WAL_USAGE UINT64CONST(0xB000000000000004)
+#define PARALLEL_KEY_BUFFER_USAGE UINT64CONST(0xB000000000000005)
+
+/*
+ * Status for index builds performed in parallel. This is allocated in a
+ * dynamic shared memory segment.
+ */
+typedef struct GinShared
+{
+ /*
+ * These fields are not modified during the build. They primarily exist
+ * for the benefit of worker processes that need to create state
+ * corresponding to that used by the leader.
+ */
+ Oid heaprelid;
+ Oid indexrelid;
+ bool isconcurrent;
+ int scantuplesortstates;
+
+ /*
+ * workersdonecv is used to monitor the progress of workers. All parallel
+ * participants must indicate that they are done before leader can use
+ * results built by the workers (and before leader can write the data into
+ * the index).
+ */
+ ConditionVariable workersdonecv;
+
+ /*
+ * mutex protects all fields before heapdesc.
+ *
+ * These fields contain status information of interest to GIN index builds
+ * that must work just the same when an index is built in parallel.
+ */
+ slock_t mutex;
+
+ /*
+ * Mutable state that is maintained by workers, and reported back to
+ * leader at end of the scans.
+ *
+ * nparticipantsdone is number of worker processes finished.
+ *
+ * reltuples is the total number of input heap tuples.
+ *
+ * indtuples is the total number of tuples that made it into the index.
+ */
+ int nparticipantsdone;
+ double reltuples;
+ double indtuples;
+
+ /*
+ * ParallelTableScanDescData data follows. Can't directly embed here, as
+ * implementations of the parallel table scan desc interface might need
+ * stronger alignment.
+ */
+} GinShared;
+
+/*
+ * Return pointer to a GinShared's parallel table scan.
+ *
+ * c.f. shm_toc_allocate as to why BUFFERALIGN is used, rather than just
+ * MAXALIGN.
+ */
+#define ParallelTableScanFromGinShared(shared) \
+ (ParallelTableScanDesc) ((char *) (shared) + BUFFERALIGN(sizeof(GinShared)))
+
+/*
+ * Status for leader in parallel index build.
+ */
+typedef struct GinLeader
+{
+ /* parallel context itself */
+ ParallelContext *pcxt;
+
+ /*
+ * nparticipanttuplesorts is the exact number of worker processes
+ * successfully launched, plus one leader process if it participates as a
+ * worker (only DISABLE_LEADER_PARTICIPATION builds avoid leader
+ * participating as a worker).
+ */
+ int nparticipanttuplesorts;
+
+ /*
+ * Leader process convenience pointers to shared state (leader avoids TOC
+ * lookups).
+ *
+ * GinShared is the shared state for entire build. sharedsort is the
+ * shared, tuplesort-managed state passed to each process tuplesort.
+ * snapshot is the snapshot used by the scan iff an MVCC snapshot is
+ * required.
+ */
+ GinShared *ginshared;
+ Sharedsort *sharedsort;
+ Snapshot snapshot;
+ WalUsage *walusage;
+ BufferUsage *bufferusage;
+} GinLeader;
typedef struct
{
@@ -32,9 +142,49 @@ typedef struct
MemoryContext tmpCtx;
MemoryContext funcCtx;
BuildAccumulator accum;
+
+ /* FIXME likely duplicate with indtuples */
+ double bs_numtuples;
+ double bs_reltuples;
+
+ /*
+ * bs_leader is only present when a parallel index build is performed, and
+ * only in the leader process. (Actually, only the leader process has a
+ * GinBuildState.)
+ */
+ GinLeader *bs_leader;
+ int bs_worker_id;
+
+ /*
+ * The sortstate is used by workers (including the leader). It has to be
+ * part of the build state, because that's the only thing passed to the
+ * build callback etc.
+ */
+ Tuplesortstate *bs_sortstate;
} GinBuildState;
+/* parallel index builds */
+static void _gin_begin_parallel(GinBuildState *buildstate, Relation heap, Relation index,
+ bool isconcurrent, int request);
+static void _gin_end_parallel(GinLeader *ginleader, GinBuildState *state);
+static Size _gin_parallel_estimate_shared(Relation heap, Snapshot snapshot);
+static double _gin_parallel_heapscan(GinBuildState *buildstate);
+static double _gin_parallel_merge(GinBuildState *buildstate);
+static void _gin_leader_participate_as_worker(GinBuildState *buildstate,
+ Relation heap, Relation index);
+static void _gin_parallel_scan_and_build(GinBuildState *buildstate,
+ GinShared *ginshared,
+ Sharedsort *sharedsort,
+ Relation heap, Relation index,
+ int sortmem, bool progress);
+
+static Datum _gin_parse_tuple(GinTuple *a, ItemPointerData **items);
+static GinTuple *_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+ Datum key, int16 typlen, bool typbyval,
+ ItemPointerData *items, uint32 nitems,
+ Size *len);
+
/*
* Adds array of item pointers to tuple's posting list, or
* creates posting tree and tuple pointing to tree in case
@@ -313,12 +463,95 @@ ginBuildCallback(Relation index, ItemPointer tid, Datum *values,
MemoryContextSwitchTo(oldCtx);
}
+static void
+ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
+ bool *isnull, bool tupleIsAlive, void *state)
+{
+ GinBuildState *buildstate = (GinBuildState *) state;
+ MemoryContext oldCtx;
+ int i;
+
+ oldCtx = MemoryContextSwitchTo(buildstate->tmpCtx);
+
+ for (i = 0; i < buildstate->ginstate.origTupdesc->natts; i++)
+ ginHeapTupleBulkInsert(buildstate, (OffsetNumber) (i + 1),
+ values[i], isnull[i], tid);
+
+ /*
+ * XXX idea - Instead of writing the entries directly into the shared
+ * tuplesort, write it into a local one, do the sort in the worker, and
+ * combine the results. For large tables with many different keys that's
+ * going to work better than the current approach where we don't get many
+ * matches in work_mem (maybe this should use 32MB, which is what we use
+ * when planning, but even that may not be great). Which means we are
+ * likely to have many entries with a single TID, forcing the leader to do
+ * a qsort() when merging the data, often amounting to ~50% of the serial
+ * part. By doing the qsort() in a worker, leader then can do a mergesort
+ * (likely cheaper). Also, it means the amount of data worker->leader is
+ * going to be lower thanks to deduplication.
+ *
+ * Disadvantage: It needs more disk space, possibly up to 2x, because each
+ * worker creates a tuplestore, then "transforms it" into the shared
+ * tuplestore (hopefully less data, but not guaranteed).
+ *
+ * It's however possible to partition the data into multiple tuplesorts
+ * per worker (by hashing). We don't need perfect sorting, and we can even
+ * live with "equal" keys having multiple hashes (if there are multiple
+ * binary representations of the value).
+ */
+
+ /*
+ * If we've maxed out our available memory, dump everything to the
+ * tuplesort
+ *
+ * XXX probably should use 32MB, not work_mem, as used during planning?
+ */
+ if (buildstate->accum.allocatedMemory >= (Size) work_mem * 1024L)
+ {
+ ItemPointerData *list;
+ Datum key;
+ GinNullCategory category;
+ uint32 nlist;
+ OffsetNumber attnum;
+ TupleDesc tdesc = RelationGetDescr(index);
+
+ ginBeginBAScan(&buildstate->accum);
+ while ((list = ginGetBAEntry(&buildstate->accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* information about the key */
+ Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
+
+ /* GIN tuple and tuple length */
+ GinTuple *tup;
+ Size tuplen;
+
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+
+ tup = _gin_build_tuple(attnum, category,
+ key, attr->attlen, attr->attbyval,
+ list, nlist, &tuplen);
+
+ tuplesort_putgintuple(buildstate->bs_sortstate, tup, tuplen);
+
+ pfree(tup);
+ }
+
+ MemoryContextReset(buildstate->tmpCtx);
+ ginInitBA(&buildstate->accum);
+ }
+
+ MemoryContextSwitchTo(oldCtx);
+}
+
IndexBuildResult *
ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
{
IndexBuildResult *result;
double reltuples;
GinBuildState buildstate;
+ GinBuildState *state = &buildstate;
Buffer RootBuffer,
MetaBuffer;
ItemPointerData *list;
@@ -336,6 +569,14 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.indtuples = 0;
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
+ /*
+ * XXX Make sure to initialize a bunch of fields, not to trip valgrind.
+ * Maybe there should be an "init" function for build state?
+ */
+ buildstate.bs_numtuples = 0;
+ buildstate.bs_reltuples = 0;
+ buildstate.bs_leader = NULL;
+
/* initialize the meta page */
MetaBuffer = GinNewBuffer(index);
@@ -376,25 +617,91 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
ginInitBA(&buildstate.accum);
/*
- * Do the heap scan. We disallow sync scan here because dataPlaceToPage
- * prefers to receive tuples in TID order.
+ * Attempt to launch parallel worker scan when required
+ *
+ * XXX plan_create_index_workers makes the number of workers dependent on
+ * maintenance_work_mem, requiring 32MB for each worker. That makes sense
+ * for btree, but not for GIN, which can do with much less memory. So
+ * maybe make that somehow less strict, optionally?
+ */
+ if (indexInfo->ii_ParallelWorkers > 0)
+ _gin_begin_parallel(state, heap, index, indexInfo->ii_Concurrent,
+ indexInfo->ii_ParallelWorkers);
+
+
+ /*
+ * If parallel build requested and at least one worker process was
+ * successfully launched, set up coordination state, wait for workers to
+ * complete. Then read all tuples from the shared tuplesort and insert
+ * them into the index.
+ *
+ * In serial mode, simply scan the table and build the index one index
+ * tuple at a time.
*/
- reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
- ginBuildCallback, (void *) &buildstate,
- NULL);
+ if (state->bs_leader)
+ {
+ SortCoordinate coordinate;
+
+ coordinate = (SortCoordinate) palloc0(sizeof(SortCoordinateData));
+ coordinate->isWorker = false;
+ coordinate->nParticipants =
+ state->bs_leader->nparticipanttuplesorts;
+ coordinate->sharedsort = state->bs_leader->sharedsort;
+
+ /*
+ * Begin leader tuplesort.
+ *
+ * In cases where parallelism is involved, the leader receives the
+ * same share of maintenance_work_mem as a serial sort (it is
+ * generally treated in the same way as a serial sort once we return).
+ * Parallel worker Tuplesortstates will have received only a fraction
+ * of maintenance_work_mem, though.
+ *
+ * We rely on the lifetime of the Leader Tuplesortstate almost not
+ * overlapping with any worker Tuplesortstate's lifetime. There may
+ * be some small overlap, but that's okay because we rely on leader
+ * Tuplesortstate only allocating a small, fixed amount of memory
+ * here. When its tuplesort_performsort() is called (by our caller),
+ * and significant amounts of memory are likely to be used, all
+ * workers must have already freed almost all memory held by their
+ * Tuplesortstates (they are about to go away completely, too). The
+ * overall effect is that maintenance_work_mem always represents an
+ * absolute high watermark on the amount of memory used by a CREATE
+ * INDEX operation, regardless of the use of parallelism or any other
+ * factor.
+ */
+ state->bs_sortstate =
+ tuplesort_begin_index_gin(maintenance_work_mem, coordinate,
+ TUPLESORT_NONE);
- /* dump remaining entries to the index */
- oldCtx = MemoryContextSwitchTo(buildstate.tmpCtx);
- ginBeginBAScan(&buildstate.accum);
- while ((list = ginGetBAEntry(&buildstate.accum,
- &attnum, &key, &category, &nlist)) != NULL)
+ /* scan the relation and merge per-worker results */
+ reltuples = _gin_parallel_merge(state);
+
+ _gin_end_parallel(state->bs_leader, state);
+ }
+ else /* no parallel index build */
{
- /* there could be many entries, so be willing to abort here */
- CHECK_FOR_INTERRUPTS();
- ginEntryInsert(&buildstate.ginstate, attnum, key, category,
- list, nlist, &buildstate.buildStats);
+ /*
+ * Do the heap scan. We disallow sync scan here because
+ * dataPlaceToPage prefers to receive tuples in TID order.
+ */
+ reltuples = table_index_build_scan(heap, index, indexInfo, false, true,
+ ginBuildCallback, (void *) &buildstate,
+ NULL);
+
+ /* dump remaining entries to the index */
+ oldCtx = MemoryContextSwitchTo(buildstate.tmpCtx);
+ ginBeginBAScan(&buildstate.accum);
+ while ((list = ginGetBAEntry(&buildstate.accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+ ginEntryInsert(&buildstate.ginstate, attnum, key, category,
+ list, nlist, &buildstate.buildStats);
+ }
+ MemoryContextSwitchTo(oldCtx);
}
- MemoryContextSwitchTo(oldCtx);
MemoryContextDelete(buildstate.funcCtx);
MemoryContextDelete(buildstate.tmpCtx);
@@ -534,3 +841,1006 @@ gininsert(Relation index, Datum *values, bool *isnull,
return false;
}
+
+/*
+ * Create parallel context, and launch workers for leader.
+ *
+ * buildstate argument should be initialized (with the exception of the
+ * tuplesort states, which may later be created based on shared
+ * state initially set up here).
+ *
+ * isconcurrent indicates if operation is CREATE INDEX CONCURRENTLY.
+ *
+ * request is the target number of parallel worker processes to launch.
+ *
+ * Sets buildstate's GinLeader, which caller must use to shut down parallel
+ * mode by passing it to _gin_end_parallel() at the very end of its index
+ * build. If not even a single worker process can be launched, this is
+ * never set, and caller should proceed with a serial index build.
+ */
+static void
+_gin_begin_parallel(GinBuildState *buildstate, Relation heap, Relation index,
+ bool isconcurrent, int request)
+{
+ ParallelContext *pcxt;
+ int scantuplesortstates;
+ Snapshot snapshot;
+ Size estginshared;
+ Size estsort;
+ GinShared *ginshared;
+ Sharedsort *sharedsort;
+ GinLeader *ginleader = (GinLeader *) palloc0(sizeof(GinLeader));
+ WalUsage *walusage;
+ BufferUsage *bufferusage;
+ bool leaderparticipates = true;
+ int querylen;
+
+#ifdef DISABLE_LEADER_PARTICIPATION
+ leaderparticipates = false;
+#endif
+
+ /*
+ * Enter parallel mode, and create context for parallel build of gin index
+ */
+ EnterParallelMode();
+ Assert(request > 0);
+ pcxt = CreateParallelContext("postgres", "_gin_parallel_build_main",
+ request);
+
+ scantuplesortstates = leaderparticipates ? request + 1 : request;
+
+ /*
+ * Prepare for scan of the base relation. In a normal index build, we use
+ * SnapshotAny because we must retrieve all tuples and do our own time
+ * qual checks (because we have to index RECENTLY_DEAD tuples). In a
+ * concurrent build, we take a regular MVCC snapshot and index whatever's
+ * live according to that.
+ */
+ if (!isconcurrent)
+ snapshot = SnapshotAny;
+ else
+ snapshot = RegisterSnapshot(GetTransactionSnapshot());
+
+ /*
+ * Estimate size for our own PARALLEL_KEY_GIN_SHARED workspace.
+ */
+ estginshared = _gin_parallel_estimate_shared(heap, snapshot);
+ shm_toc_estimate_chunk(&pcxt->estimator, estginshared);
+ estsort = tuplesort_estimate_shared(scantuplesortstates);
+ shm_toc_estimate_chunk(&pcxt->estimator, estsort);
+
+ shm_toc_estimate_keys(&pcxt->estimator, 2);
+
+ /*
+ * Estimate space for WalUsage and BufferUsage -- PARALLEL_KEY_WAL_USAGE
+ * and PARALLEL_KEY_BUFFER_USAGE.
+ *
+ * If there are no extensions loaded that care, we could skip this. We
+ * have no way of knowing whether anyone's looking at pgWalUsage or
+ * pgBufferUsage, so do it unconditionally.
+ */
+ shm_toc_estimate_chunk(&pcxt->estimator,
+ mul_size(sizeof(WalUsage), pcxt->nworkers));
+ shm_toc_estimate_keys(&pcxt->estimator, 1);
+ shm_toc_estimate_chunk(&pcxt->estimator,
+ mul_size(sizeof(BufferUsage), pcxt->nworkers));
+ shm_toc_estimate_keys(&pcxt->estimator, 1);
+
+ /* Finally, estimate PARALLEL_KEY_QUERY_TEXT space */
+ if (debug_query_string)
+ {
+ querylen = strlen(debug_query_string);
+ shm_toc_estimate_chunk(&pcxt->estimator, querylen + 1);
+ shm_toc_estimate_keys(&pcxt->estimator, 1);
+ }
+ else
+ querylen = 0; /* keep compiler quiet */
+
+ /* Everyone's had a chance to ask for space, so now create the DSM */
+ InitializeParallelDSM(pcxt);
+
+ /* If no DSM segment was available, back out (do serial build) */
+ if (pcxt->seg == NULL)
+ {
+ if (IsMVCCSnapshot(snapshot))
+ UnregisterSnapshot(snapshot);
+ DestroyParallelContext(pcxt);
+ ExitParallelMode();
+ return;
+ }
+
+ /* Store shared build state, for which we reserved space */
+ ginshared = (GinShared *) shm_toc_allocate(pcxt->toc, estginshared);
+ /* Initialize immutable state */
+ ginshared->heaprelid = RelationGetRelid(heap);
+ ginshared->indexrelid = RelationGetRelid(index);
+ ginshared->isconcurrent = isconcurrent;
+ ginshared->scantuplesortstates = scantuplesortstates;
+
+ ConditionVariableInit(&ginshared->workersdonecv);
+ SpinLockInit(&ginshared->mutex);
+
+ /* Initialize mutable state */
+ ginshared->nparticipantsdone = 0;
+ ginshared->reltuples = 0.0;
+ ginshared->indtuples = 0.0;
+
+ table_parallelscan_initialize(heap,
+ ParallelTableScanFromGinShared(ginshared),
+ snapshot);
+
+ /*
+ * Store shared tuplesort-private state, for which we reserved space.
+ * Then, initialize opaque state using tuplesort routine.
+ */
+ sharedsort = (Sharedsort *) shm_toc_allocate(pcxt->toc, estsort);
+ tuplesort_initialize_shared(sharedsort, scantuplesortstates,
+ pcxt->seg);
+
+ /*
+ * Store shared tuplesort-private state, for which we reserved space.
+ * Then, initialize opaque state using tuplesort routine.
+ */
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_GIN_SHARED, ginshared);
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_TUPLESORT, sharedsort);
+
+ /* Store query string for workers */
+ if (debug_query_string)
+ {
+ char *sharedquery;
+
+ sharedquery = (char *) shm_toc_allocate(pcxt->toc, querylen + 1);
+ memcpy(sharedquery, debug_query_string, querylen + 1);
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_QUERY_TEXT, sharedquery);
+ }
+
+ /*
+ * Allocate space for each worker's WalUsage and BufferUsage; no need to
+ * initialize.
+ */
+ walusage = shm_toc_allocate(pcxt->toc,
+ mul_size(sizeof(WalUsage), pcxt->nworkers));
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_WAL_USAGE, walusage);
+ bufferusage = shm_toc_allocate(pcxt->toc,
+ mul_size(sizeof(BufferUsage), pcxt->nworkers));
+ shm_toc_insert(pcxt->toc, PARALLEL_KEY_BUFFER_USAGE, bufferusage);
+
+ /* Launch workers, saving status for leader/caller */
+ LaunchParallelWorkers(pcxt);
+ ginleader->pcxt = pcxt;
+ ginleader->nparticipanttuplesorts = pcxt->nworkers_launched;
+ if (leaderparticipates)
+ ginleader->nparticipanttuplesorts++;
+ ginleader->ginshared = ginshared;
+ ginleader->sharedsort = sharedsort;
+ ginleader->snapshot = snapshot;
+ ginleader->walusage = walusage;
+ ginleader->bufferusage = bufferusage;
+
+ /* If no workers were successfully launched, back out (do serial build) */
+ if (pcxt->nworkers_launched == 0)
+ {
+ _gin_end_parallel(ginleader, NULL);
+ return;
+ }
+
+ /* Save leader state now that it's clear build will be parallel */
+ buildstate->bs_leader = ginleader;
+
+ /* Join heap scan ourselves */
+ if (leaderparticipates)
+ _gin_leader_participate_as_worker(buildstate, heap, index);
+
+ /*
+ * Caller needs to wait for all launched workers when we return. Make
+ * sure that the failure-to-start case will not hang forever.
+ */
+ WaitForParallelWorkersToAttach(pcxt);
+}
+
+/*
+ * Shut down workers, destroy parallel context, and end parallel mode.
+ */
+static void
+_gin_end_parallel(GinLeader *ginleader, GinBuildState *state)
+{
+ int i;
+
+ /* Shutdown worker processes */
+ WaitForParallelWorkersToFinish(ginleader->pcxt);
+
+ /*
+ * Next, accumulate WAL usage. (This must wait for the workers to finish,
+ * or we might get incomplete data.)
+ */
+ for (i = 0; i < ginleader->pcxt->nworkers_launched; i++)
+ InstrAccumParallelQuery(&ginleader->bufferusage[i], &ginleader->walusage[i]);
+
+ /* Free last reference to MVCC snapshot, if one was used */
+ if (IsMVCCSnapshot(ginleader->snapshot))
+ UnregisterSnapshot(ginleader->snapshot);
+ DestroyParallelContext(ginleader->pcxt);
+ ExitParallelMode();
+}
+
+/*
+ * Within leader, wait for end of heap scan.
+ *
+ * When called, parallel heap scan started by _gin_begin_parallel() will
+ * already be underway within worker processes (when leader participates
+ * as a worker, we should end up here just as workers are finishing).
+ *
+ * Returns the total number of heap tuples scanned.
+ */
+static double
+_gin_parallel_heapscan(GinBuildState *state)
+{
+ GinShared *ginshared = state->bs_leader->ginshared;
+ int nparticipanttuplesorts;
+
+ nparticipanttuplesorts = state->bs_leader->nparticipanttuplesorts;
+ for (;;)
+ {
+ SpinLockAcquire(&ginshared->mutex);
+ if (ginshared->nparticipantsdone == nparticipanttuplesorts)
+ {
+ /* copy the data into leader state */
+ state->bs_reltuples = ginshared->reltuples;
+ state->bs_numtuples = ginshared->indtuples;
+
+ SpinLockRelease(&ginshared->mutex);
+ break;
+ }
+ SpinLockRelease(&ginshared->mutex);
+
+ ConditionVariableSleep(&ginshared->workersdonecv,
+ WAIT_EVENT_PARALLEL_CREATE_INDEX_SCAN);
+ }
+
+ ConditionVariableCancelSleep();
+
+ return state->bs_reltuples;
+}
+
+static int
+tid_cmp(const void *a, const void *b)
+{
+ return ItemPointerCompare((ItemPointer) a, (ItemPointer) b);
+}
+
+/*
+ * State used to combine accumulate TIDs from multiple GinTuples for the same
+ * key value.
+ *
+ * XXX Similar purpose to BuildAccumulator, but much simpler.
+ */
+typedef struct GinBuffer
+{
+ OffsetNumber attnum;
+ GinNullCategory category;
+ Datum key; /* 0 if no key (and keylen == 0) */
+ Size keylen; /* number of bytes (not typlen) */
+
+ /* type info */
+ int16 typlen;
+ bool typbyval;
+
+ /* array of TID values */
+ int nitems;
+ int maxitems;
+ ItemPointerData *items;
+} GinBuffer;
+
+/* XXX should do more checks */
+static void
+AssertCheckGinBuffer(GinBuffer *buffer)
+{
+#ifdef USE_ASSERT_CHECKING
+ Assert(buffer->nitems <= buffer->maxitems);
+#endif
+}
+
+static void
+AssertCheckItemPointers(ItemPointerData *items, int nitems, bool sorted)
+{
+#ifdef USE_ASSERT_CHECKING
+ for (int i = 0; i < nitems; i++)
+ {
+ Assert(ItemPointerIsValid(&items[i]));
+
+ if ((i == 0) || !sorted)
+ continue;
+
+ Assert(ItemPointerCompare(&items[i - 1], &items[i]) < 0);
+ }
+#endif
+}
+
+static GinBuffer *
+GinBufferInit(void)
+{
+ return palloc0(sizeof(GinBuffer));
+}
+
+static bool
+GinBufferIsEmpty(GinBuffer *buffer)
+{
+ return (buffer->nitems == 0);
+}
+
+/*
+ * Compare if the tuple matches the already accumulated data. Compare
+ * scalar fields first, before the actual key.
+ *
+ * XXX The key is compared using memcmp, which means that if a key has
+ * multiple binary representations, we may end up treating them as
+ * different here. But that's OK, the index will merge them anyway.
+ */
+static bool
+GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
+{
+ AssertCheckGinBuffer(buffer);
+
+ if (tup->attrnum != buffer->attnum)
+ return false;
+
+ /* same attribute should have the same type info */
+ Assert(tup->typbyval == buffer->typbyval);
+ Assert(tup->typlen == buffer->typlen);
+
+ if (tup->category != buffer->category)
+ return false;
+
+ if (tup->keylen != buffer->keylen)
+ return false;
+
+ /*
+ * For NULL/empty keys, this means equality, for normal keys we need to
+ * compare the actual key value.
+ */
+ if (buffer->category != GIN_CAT_NORM_KEY)
+ return true;
+
+ /*
+ * Compare the key value, depending on the type information.
+ *
+ * XXX Not sure this works correctly for byval types that don't need the
+ * whole Datum. What if there is garbage in the padding bytes?
+ */
+ if (buffer->typbyval)
+ return (buffer->key == *(Datum *) tup->data);
+
+ /* byref values simply uses memcmp for comparison */
+ return (memcmp(tup->data, DatumGetPointer(buffer->key), buffer->keylen) == 0);
+}
+
+static void
+GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
+{
+ ItemPointerData *items;
+ Datum key;
+
+ AssertCheckGinBuffer(buffer);
+
+ key = _gin_parse_tuple(tup, &items);
+
+ /* if the buffer is empty, set the fields (and copy the key) */
+ if (GinBufferIsEmpty(buffer))
+ {
+ buffer->category = tup->category;
+ buffer->keylen = tup->keylen;
+ buffer->attnum = tup->attrnum;
+
+ buffer->typlen = tup->typlen;
+ buffer->typbyval = tup->typbyval;
+
+ if (tup->category == GIN_CAT_NORM_KEY)
+ buffer->key = datumCopy(key, buffer->typbyval, buffer->typlen);
+ else
+ buffer->key = (Datum) 0;
+ }
+
+ /* enlarge the TID buffer, if needed */
+ if (buffer->nitems + tup->nitems > buffer->maxitems)
+ {
+ /* 64 seems like a good init value */
+ buffer->maxitems = Max(buffer->maxitems, 64);
+
+ while (buffer->nitems + tup->nitems > buffer->maxitems)
+ buffer->maxitems *= 2;
+
+ if (buffer->items == NULL)
+ buffer->items = palloc(buffer->maxitems * sizeof(ItemPointerData));
+ else
+ buffer->items = repalloc(buffer->items,
+ buffer->maxitems * sizeof(ItemPointerData));
+ }
+
+ /* now we should be guaranteed to have enough space for all the TIDs */
+ Assert(buffer->nitems + tup->nitems <= buffer->maxitems);
+
+ /* copy the new TIDs into the buffer */
+ memcpy(&buffer->items[buffer->nitems], items, sizeof(ItemPointerData) * tup->nitems);
+ buffer->nitems += tup->nitems;
+
+ AssertCheckItemPointers(buffer->items, buffer->nitems, false);
+}
+
+static void
+GinBufferSortItems(GinBuffer *buffer)
+{
+ /* we should not have a buffer with no TIDs to sort */
+ Assert(buffer->items != NULL);
+ Assert(buffer->nitems > 0);
+
+ pg_qsort(buffer->items, buffer->nitems, sizeof(ItemPointerData), tid_cmp);
+
+ AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+}
+
+/* XXX probably would be better to have a memory context for the buffer */
+static void
+GinBufferReset(GinBuffer *buffer)
+{
+ Assert(!GinBufferIsEmpty(buffer));
+
+ /* release byref values, do nothing for by-val ones */
+ if ((buffer->category == GIN_CAT_NORM_KEY) && !buffer->typbyval)
+ pfree(DatumGetPointer(buffer->key));
+
+ /* XXX not really needed, but easier to trigger NULL deref etc. */
+ buffer->key = (Datum) 0;
+
+ buffer->attnum = 0;
+ buffer->category = 0;
+ buffer->keylen = 0;
+ buffer->nitems = 0;
+
+ buffer->typlen = 0;
+ buffer->typbyval = 0;
+
+ /* XXX should do something with extremely large array of items? */
+}
+
+/*
+ * XXX Maybe check size of the TID arrays, and return false if it's too
+ * large (more thant maintenance_work_mem or something?).
+ */
+static bool
+GinBufferCanAddKey(GinBuffer *buffer, GinTuple *tup)
+{
+ /* empty buffer can accept data for any key */
+ if (GinBufferIsEmpty(buffer))
+ return true;
+
+ /* otherwise just data for the same key */
+ return GinBufferKeyEquals(buffer, tup);
+}
+
+/*
+ * Within leader, wait for end of heap scan and merge per-worker results.
+ *
+ * After waiting for all workers to finish, merge the per-worker results into
+ * the complete index. The results from each worker are sorted by block number
+ * (start of the page range). While combinig the per-worker results we merge
+ * summaries for the same page range, and also fill-in empty summaries for
+ * ranges without any tuples.
+ *
+ * Returns the total number of heap tuples scanned.
+ *
+ * FIXME probably should have local memory contexts similar to what
+ * _brin_parallel_merge does.
+ */
+static double
+_gin_parallel_merge(GinBuildState *state)
+{
+ GinTuple *tup;
+ Size tuplen;
+ double reltuples = 0;
+ GinBuffer *buffer;
+
+ /* wait for workers to scan table and produce partial results */
+ reltuples = _gin_parallel_heapscan(state);
+
+ /* do the actual sort in the leader */
+ tuplesort_performsort(state->bs_sortstate);
+
+ /* initialize buffer to combine entries for the same key */
+ buffer = GinBufferInit();
+
+ /*
+ * Read the GIN tuples from the shared tuplesort, sorted by category and
+ * key. That probably gives us order matching how data is organized in the
+ * index.
+ *
+ * XXX Maybe we should sort by key first, then by category?
+ *
+ * We don't insert the GIN tuples right away, but instead accumulate as
+ * many TIDs for the same key as possible, and then insert that at once.
+ * This way we don't need to decompress/recompress the posting lists, etc.
+ */
+ while ((tup = tuplesort_getgintuple(state->bs_sortstate, &tuplen, true)) != NULL)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * If the buffer can accept the new GIN tuple, just store it there and
+ * we're done. If it's a different key (or maybe too much data) flush
+ * the current contents into the index first.
+ */
+ if (!GinBufferCanAddKey(buffer, tup))
+ {
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ GinBufferSortItems(buffer);
+
+ ginEntryInsert(&state->ginstate,
+ buffer->attnum, buffer->key, buffer->category,
+ buffer->items, buffer->nitems, &state->buildStats);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ /* now remember the new key */
+ GinBufferStoreTuple(buffer, tup);
+ }
+
+ /* flush data remaining in the buffer (for the last key) */
+ if (!GinBufferIsEmpty(buffer))
+ {
+ GinBufferSortItems(buffer);
+
+ ginEntryInsert(&state->ginstate,
+ buffer->attnum, buffer->key, buffer->category,
+ buffer->items, buffer->nitems, &state->buildStats);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ tuplesort_end(state->bs_sortstate);
+
+ return reltuples;
+}
+
+/*
+ * Returns size of shared memory required to store state for a parallel
+ * gin index build based on the snapshot its parallel scan will use.
+ */
+static Size
+_gin_parallel_estimate_shared(Relation heap, Snapshot snapshot)
+{
+ /* c.f. shm_toc_allocate as to why BUFFERALIGN is used */
+ return add_size(BUFFERALIGN(sizeof(GinShared)),
+ table_parallelscan_estimate(heap, snapshot));
+}
+
+/*
+ * Within leader, participate as a parallel worker.
+ */
+static void
+_gin_leader_participate_as_worker(GinBuildState *buildstate, Relation heap, Relation index)
+{
+ GinLeader *ginleader = buildstate->bs_leader;
+ int sortmem;
+
+ /*
+ * Might as well use reliable figure when doling out maintenance_work_mem
+ * (when requested number of workers were not launched, this will be
+ * somewhat higher than it is for other workers).
+ */
+ sortmem = maintenance_work_mem / ginleader->nparticipanttuplesorts;
+
+ /* Perform work common to all participants */
+ _gin_parallel_scan_and_build(buildstate, ginleader->ginshared,
+ ginleader->sharedsort, heap, index, sortmem, true);
+}
+
+/*
+ * Perform a worker's portion of a parallel sort.
+ *
+ * This generates a tuplesort for the worker portion of the table.
+ *
+ * sortmem is the amount of working memory to use within each worker,
+ * expressed in KBs.
+ *
+ * When this returns, workers are done, and need only release resources.
+ */
+static void
+_gin_parallel_scan_and_build(GinBuildState *state,
+ GinShared *ginshared, Sharedsort *sharedsort,
+ Relation heap, Relation index,
+ int sortmem, bool progress)
+{
+ SortCoordinate coordinate;
+ TableScanDesc scan;
+ double reltuples;
+ IndexInfo *indexInfo;
+
+ /* Initialize local tuplesort coordination state */
+ coordinate = palloc0(sizeof(SortCoordinateData));
+ coordinate->isWorker = true;
+ coordinate->nParticipants = -1;
+ coordinate->sharedsort = sharedsort;
+
+ /* Begin "partial" tuplesort */
+ state->bs_sortstate = tuplesort_begin_index_gin(sortmem, coordinate,
+ TUPLESORT_NONE);
+
+ /* Join parallel scan */
+ indexInfo = BuildIndexInfo(index);
+ indexInfo->ii_Concurrent = ginshared->isconcurrent;
+
+ scan = table_beginscan_parallel(heap,
+ ParallelTableScanFromGinShared(ginshared));
+
+ reltuples = table_index_build_scan(heap, index, indexInfo, true, true,
+ ginBuildCallbackParallel, state, scan);
+
+ /* insert the last item */
+ /* write remaining accumulated entries */
+ {
+ ItemPointerData *list;
+ Datum key;
+ GinNullCategory category;
+ uint32 nlist;
+ OffsetNumber attnum;
+ TupleDesc tdesc = RelationGetDescr(index);
+
+ ginBeginBAScan(&state->accum);
+ while ((list = ginGetBAEntry(&state->accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* information about the key */
+ Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
+
+ GinTuple *tup;
+ Size len;
+
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+
+ tup = _gin_build_tuple(attnum, category,
+ key, attr->attlen, attr->attbyval,
+ list, nlist, &len);
+
+ tuplesort_putgintuple(state->bs_sortstate, tup, len);
+
+ pfree(tup);
+ }
+
+ MemoryContextReset(state->tmpCtx);
+ ginInitBA(&state->accum);
+ }
+
+ /* sort the GIN tuples built by this worker */
+ tuplesort_performsort(state->bs_sortstate);
+
+ state->bs_reltuples += reltuples;
+
+ /*
+ * Done. Record ambuild statistics.
+ */
+ SpinLockAcquire(&ginshared->mutex);
+ ginshared->nparticipantsdone++;
+ ginshared->reltuples += state->bs_reltuples;
+ ginshared->indtuples += state->bs_numtuples;
+ SpinLockRelease(&ginshared->mutex);
+
+ /* Notify leader */
+ ConditionVariableSignal(&ginshared->workersdonecv);
+
+ tuplesort_end(state->bs_sortstate);
+}
+
+/*
+ * Perform work within a launched parallel process.
+ */
+void
+_gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
+{
+ char *sharedquery;
+ GinShared *ginshared;
+ Sharedsort *sharedsort;
+ GinBuildState buildstate;
+ Relation heapRel;
+ Relation indexRel;
+ LOCKMODE heapLockmode;
+ LOCKMODE indexLockmode;
+ WalUsage *walusage;
+ BufferUsage *bufferusage;
+ int sortmem;
+
+ /*
+ * The only possible status flag that can be set to the parallel worker is
+ * PROC_IN_SAFE_IC.
+ */
+ Assert((MyProc->statusFlags == 0) ||
+ (MyProc->statusFlags == PROC_IN_SAFE_IC));
+
+ /* Set debug_query_string for individual workers first */
+ sharedquery = shm_toc_lookup(toc, PARALLEL_KEY_QUERY_TEXT, true);
+ debug_query_string = sharedquery;
+
+ /* Report the query string from leader */
+ pgstat_report_activity(STATE_RUNNING, debug_query_string);
+
+ /* Look up gin shared state */
+ ginshared = shm_toc_lookup(toc, PARALLEL_KEY_GIN_SHARED, false);
+
+ /* Open relations using lock modes known to be obtained by index.c */
+ if (!ginshared->isconcurrent)
+ {
+ heapLockmode = ShareLock;
+ indexLockmode = AccessExclusiveLock;
+ }
+ else
+ {
+ heapLockmode = ShareUpdateExclusiveLock;
+ indexLockmode = RowExclusiveLock;
+ }
+
+ /* Open relations within worker */
+ heapRel = table_open(ginshared->heaprelid, heapLockmode);
+ indexRel = index_open(ginshared->indexrelid, indexLockmode);
+
+ /* initialize the GIN build state */
+ initGinState(&buildstate.ginstate, indexRel);
+ buildstate.indtuples = 0;
+ memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
+
+ /*
+ * create a temporary memory context that is used to hold data not yet
+ * dumped out to the index
+ */
+ buildstate.tmpCtx = AllocSetContextCreate(CurrentMemoryContext,
+ "Gin build temporary context",
+ ALLOCSET_DEFAULT_SIZES);
+
+ /*
+ * create a temporary memory context that is used for calling
+ * ginExtractEntries(), and can be reset after each tuple
+ */
+ buildstate.funcCtx = AllocSetContextCreate(CurrentMemoryContext,
+ "Gin build temporary context for user-defined function",
+ ALLOCSET_DEFAULT_SIZES);
+
+ buildstate.accum.ginstate = &buildstate.ginstate;
+ ginInitBA(&buildstate.accum);
+
+
+ /* Look up shared state private to tuplesort.c */
+ sharedsort = shm_toc_lookup(toc, PARALLEL_KEY_TUPLESORT, false);
+ tuplesort_attach_shared(sharedsort, seg);
+
+ /* Prepare to track buffer usage during parallel execution */
+ InstrStartParallelQuery();
+
+ /*
+ * Might as well use reliable figure when doling out maintenance_work_mem
+ * (when requested number of workers were not launched, this will be
+ * somewhat higher than it is for other workers).
+ */
+ sortmem = maintenance_work_mem / ginshared->scantuplesortstates;
+
+ _gin_parallel_scan_and_build(&buildstate, ginshared, sharedsort,
+ heapRel, indexRel, sortmem, false);
+
+ /* Report WAL/buffer usage during parallel execution */
+ bufferusage = shm_toc_lookup(toc, PARALLEL_KEY_BUFFER_USAGE, false);
+ walusage = shm_toc_lookup(toc, PARALLEL_KEY_WAL_USAGE, false);
+ InstrEndParallelQuery(&bufferusage[ParallelWorkerNumber],
+ &walusage[ParallelWorkerNumber]);
+
+ index_close(indexRel, indexLockmode);
+ table_close(heapRel, heapLockmode);
+}
+
+/*
+ * _gin_build_tuple
+ * Serialize the state for an index key into a tuple for tuplesort.
+ *
+ * The tuple has a number of scalar fields (mostly matching the build state),
+ * and then a data array that stores the key first, and then the TID list.
+ *
+ * For by-reference data types, we store the actual data. For by-val types
+ * we simply copy the whole Datum, so that we don't have to care about stuff
+ * like endianess etc. We could make it a little bit smaller, but it's not
+ * worth it - it's a tiny fraction of the data, and we need to MAXALIGN the
+ * start of the TID list anyway. So we wouldn't save anything.
+ */
+static GinTuple *
+_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+ Datum key, int16 typlen, bool typbyval,
+ ItemPointerData *items, uint32 nitems,
+ Size *len)
+{
+ GinTuple *tuple;
+ char *ptr;
+
+ Size tuplen;
+ int keylen;
+
+ /*
+ * Calculate how long is the key value. Only keys with GIN_CAT_NORM_KEY
+ * have actual non-empty key. We include varlena headers and \0 bytes for
+ * strings, to make it easier to access the data in-line.
+ *
+ * For byval types we simply copy the whole Datum. We could store just the
+ * necessary bytes, but this is simpler to work with and not worth the
+ * extra complexity. Moreover we still need to do the MAXALIGN to allow
+ * direct access to items pointers.
+ */
+ if (category != GIN_CAT_NORM_KEY)
+ keylen = 0;
+ else if (typbyval)
+ keylen = sizeof(Datum);
+ else if (typlen > 0)
+ keylen = typlen;
+ else if (typlen == -1)
+ keylen = VARSIZE_ANY(key);
+ else if (typlen == -2)
+ keylen = strlen(DatumGetPointer(key)) + 1;
+ else
+ elog(ERROR, "invalid typlen");
+
+ /*
+ * Determine GIN tuple length with all the data included. Be careful about
+ * alignment, to allow direct access to item pointers.
+ */
+ tuplen = MAXALIGN(offsetof(GinTuple, data) + keylen) +
+ (sizeof(ItemPointerData) * nitems);
+
+ *len = tuplen;
+
+ /*
+ * allocate space for the whole GIN tuple
+ *
+ * XXX palloc0 so that valgrind does not complain about uninitialized
+ * bytes in writetup_index_gin, likely because of padding
+ */
+ tuple = palloc0(tuplen);
+
+ tuple->tuplen = tuplen;
+ tuple->attrnum = attrnum;
+ tuple->category = category;
+ tuple->keylen = keylen;
+ tuple->nitems = nitems;
+
+ /* key type info */
+ tuple->typlen = typlen;
+ tuple->typbyval = typbyval;
+
+ /*
+ * Copy the key and items into the tuple. First the key value, which we
+ * can simply copy right at the beginning of the data array.
+ */
+ if (category == GIN_CAT_NORM_KEY)
+ {
+ if (typbyval)
+ {
+ memcpy(tuple->data, &key, sizeof(Datum));
+ }
+ else if (typlen > 0) /* byref, fixed length */
+ {
+ memcpy(tuple->data, DatumGetPointer(key), typlen);
+ }
+ else if (typlen == -1)
+ {
+ memcpy(tuple->data, DatumGetPointer(key), keylen);
+ }
+ else if (typlen == -2)
+ {
+ memcpy(tuple->data, DatumGetPointer(key), keylen);
+ }
+ }
+
+ /* finally, copy the TIDs into the array */
+ ptr = (char *) tuple + MAXALIGN(offsetof(GinTuple, data) + keylen);
+
+ memcpy(ptr, items, sizeof(ItemPointerData) * nitems);
+
+ return tuple;
+}
+
+/*
+ * _gin_parse_tuple
+ * Deserialize the tuple from the tuplestore representation.
+ *
+ * Most of the fields are actually directly accessible, the only thing that
+ * needs more care is the key and the TID list.
+ *
+ * For the key, this returns a regular Datum representing it. It's either the
+ * actual key value, or a pointer to the beginning of the data array (which is
+ * where the data was copied by _gin_build_tuple).
+ *
+ * The pointer to the TID list is returned through 'items' (which is simply
+ * a pointer to the data array).
+ */
+static Datum
+_gin_parse_tuple(GinTuple *a, ItemPointerData **items)
+{
+ Datum key;
+
+ if (items)
+ {
+ char *ptr = (char *) a + MAXALIGN(offsetof(GinTuple, data) + a->keylen);
+
+ *items = (ItemPointerData *) ptr;
+ }
+
+ if (a->category != GIN_CAT_NORM_KEY)
+ return (Datum) 0;
+
+ if (a->typbyval)
+ {
+ memcpy(&key, a->data, a->keylen);
+ return key;
+ }
+
+ return PointerGetDatum(a->data);
+}
+
+/*
+ * _gin_compare_tuples
+ * Compare GIN tuples, used by tuplesort during parallel index build.
+ *
+ * The scalar fields (attrnum, category) are compared first, the key value is
+ * compared last. The comparisons are done simply by "memcmp", based on the
+ * assumption that if we get two keys that are two different representations
+ * of a logically equal value, it'll get merged by the index build.
+ *
+ * FIXME Is the assumption we can just memcmp() actually valid? Won't this
+ * trigger the "could not split GIN page; all old items didn't fit" error
+ * when trying to update the TID list?
+ */
+int
+_gin_compare_tuples(GinTuple *a, GinTuple *b)
+{
+ Datum keya,
+ keyb;
+
+ if (a->attrnum < b->attrnum)
+ return -1;
+
+ if (a->attrnum > b->attrnum)
+ return 1;
+
+ if (a->category < b->category)
+ return -1;
+
+ if (a->category > b->category)
+ return 1;
+
+ if ((a->category == GIN_CAT_NORM_KEY) &&
+ (b->category == GIN_CAT_NORM_KEY))
+ {
+ keya = _gin_parse_tuple(a, NULL);
+ keyb = _gin_parse_tuple(b, NULL);
+
+ /*
+ * works for both byval and byref types with fixed lenght, because for
+ * byval we set keylen to sizeof(Datum)
+ */
+ if (a->typbyval)
+ {
+ return memcmp(&keya, &keyb, a->keylen);
+ }
+ else
+ {
+ if (a->keylen < b->keylen)
+ return -1;
+
+ if (a->keylen > b->keylen)
+ return 1;
+
+ return memcmp(DatumGetPointer(keya), DatumGetPointer(keyb), a->keylen);
+ }
+ }
+
+ return 0;
+}
diff --git a/src/backend/access/gin/ginutil.c b/src/backend/access/gin/ginutil.c
index 5747ae6a4ca..dd22b44aca9 100644
--- a/src/backend/access/gin/ginutil.c
+++ b/src/backend/access/gin/ginutil.c
@@ -53,7 +53,7 @@ ginhandler(PG_FUNCTION_ARGS)
amroutine->amclusterable = false;
amroutine->ampredlocks = true;
amroutine->amcanparallel = false;
- amroutine->amcanbuildparallel = false;
+ amroutine->amcanbuildparallel = true;
amroutine->amcaninclude = false;
amroutine->amusemaintenanceworkmem = true;
amroutine->amsummarizing = false;
diff --git a/src/backend/access/transam/parallel.c b/src/backend/access/transam/parallel.c
index 8613fc6fb54..c9ea769afb5 100644
--- a/src/backend/access/transam/parallel.c
+++ b/src/backend/access/transam/parallel.c
@@ -15,6 +15,7 @@
#include "postgres.h"
#include "access/brin.h"
+#include "access/gin.h"
#include "access/nbtree.h"
#include "access/parallel.h"
#include "access/session.h"
@@ -146,6 +147,9 @@ static const struct
{
"_brin_parallel_build_main", _brin_parallel_build_main
},
+ {
+ "_gin_parallel_build_main", _gin_parallel_build_main
+ },
{
"parallel_vacuum_main", parallel_vacuum_main
}
diff --git a/src/backend/utils/sort/tuplesortvariants.c b/src/backend/utils/sort/tuplesortvariants.c
index 05a853caa36..55cc55969e5 100644
--- a/src/backend/utils/sort/tuplesortvariants.c
+++ b/src/backend/utils/sort/tuplesortvariants.c
@@ -20,6 +20,7 @@
#include "postgres.h"
#include "access/brin_tuple.h"
+#include "access/gin_tuple.h"
#include "access/hash.h"
#include "access/htup_details.h"
#include "access/nbtree.h"
@@ -46,6 +47,8 @@ static void removeabbrev_index(Tuplesortstate *state, SortTuple *stups,
int count);
static void removeabbrev_index_brin(Tuplesortstate *state, SortTuple *stups,
int count);
+static void removeabbrev_index_gin(Tuplesortstate *state, SortTuple *stups,
+ int count);
static void removeabbrev_datum(Tuplesortstate *state, SortTuple *stups,
int count);
static int comparetup_heap(const SortTuple *a, const SortTuple *b,
@@ -74,6 +77,8 @@ static int comparetup_index_hash_tiebreak(const SortTuple *a, const SortTuple *b
Tuplesortstate *state);
static int comparetup_index_brin(const SortTuple *a, const SortTuple *b,
Tuplesortstate *state);
+static int comparetup_index_gin(const SortTuple *a, const SortTuple *b,
+ Tuplesortstate *state);
static void writetup_index(Tuplesortstate *state, LogicalTape *tape,
SortTuple *stup);
static void readtup_index(Tuplesortstate *state, SortTuple *stup,
@@ -82,6 +87,10 @@ static void writetup_index_brin(Tuplesortstate *state, LogicalTape *tape,
SortTuple *stup);
static void readtup_index_brin(Tuplesortstate *state, SortTuple *stup,
LogicalTape *tape, unsigned int len);
+static void writetup_index_gin(Tuplesortstate *state, LogicalTape *tape,
+ SortTuple *stup);
+static void readtup_index_gin(Tuplesortstate *state, SortTuple *stup,
+ LogicalTape *tape, unsigned int len);
static int comparetup_datum(const SortTuple *a, const SortTuple *b,
Tuplesortstate *state);
static int comparetup_datum_tiebreak(const SortTuple *a, const SortTuple *b,
@@ -580,6 +589,35 @@ tuplesort_begin_index_brin(int workMem,
return state;
}
+
+Tuplesortstate *
+tuplesort_begin_index_gin(int workMem, SortCoordinate coordinate,
+ int sortopt)
+{
+ Tuplesortstate *state = tuplesort_begin_common(workMem, coordinate,
+ sortopt);
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+
+#ifdef TRACE_SORT
+ if (trace_sort)
+ elog(LOG,
+ "begin index sort: workMem = %d, randomAccess = %c",
+ workMem,
+ sortopt & TUPLESORT_RANDOMACCESS ? 't' : 'f');
+#endif
+
+ base->nKeys = 1; /* Only the index key */
+
+ base->removeabbrev = removeabbrev_index_gin;
+ base->comparetup = comparetup_index_gin;
+ base->writetup = writetup_index_gin;
+ base->readtup = readtup_index_gin;
+ base->haveDatum1 = false;
+ base->arg = NULL;
+
+ return state;
+}
+
Tuplesortstate *
tuplesort_begin_datum(Oid datumType, Oid sortOperator, Oid sortCollation,
bool nullsFirstFlag, int workMem,
@@ -817,6 +855,37 @@ tuplesort_putbrintuple(Tuplesortstate *state, BrinTuple *tuple, Size size)
MemoryContextSwitchTo(oldcontext);
}
+void
+tuplesort_putgintuple(Tuplesortstate *state, GinTuple *tup, Size size)
+{
+ SortTuple stup;
+ GinTuple *ctup;
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ MemoryContext oldcontext = MemoryContextSwitchTo(base->tuplecontext);
+ Size tuplen;
+
+ /* copy the GinTuple into the right memory context */
+ ctup = palloc(size);
+ memcpy(ctup, tup, size);
+
+ stup.tuple = ctup;
+ stup.datum1 = (Datum) 0;
+ stup.isnull1 = false;
+
+ /* GetMemoryChunkSpace is not supported for bump contexts */
+ if (TupleSortUseBumpTupleCxt(base->sortopt))
+ tuplen = MAXALIGN(size);
+ else
+ tuplen = GetMemoryChunkSpace(ctup);
+
+ tuplesort_puttuple_common(state, &stup,
+ base->sortKeys &&
+ base->sortKeys->abbrev_converter &&
+ !stup.isnull1, tuplen);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Accept one Datum while collecting input data for sort.
*
@@ -989,6 +1058,29 @@ tuplesort_getbrintuple(Tuplesortstate *state, Size *len, bool forward)
return &btup->tuple;
}
+GinTuple *
+tuplesort_getgintuple(Tuplesortstate *state, Size *len, bool forward)
+{
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ MemoryContext oldcontext = MemoryContextSwitchTo(base->sortcontext);
+ SortTuple stup;
+ GinTuple *tup;
+
+ if (!tuplesort_gettuple_common(state, forward, &stup))
+ stup.tuple = NULL;
+
+ MemoryContextSwitchTo(oldcontext);
+
+ if (!stup.tuple)
+ return false;
+
+ tup = (GinTuple *) stup.tuple;
+
+ *len = tup->tuplen;
+
+ return tup;
+}
+
/*
* Fetch the next Datum in either forward or back direction.
* Returns false if no more datums.
@@ -1777,6 +1869,68 @@ readtup_index_brin(Tuplesortstate *state, SortTuple *stup,
stup->datum1 = tuple->tuple.bt_blkno;
}
+
+/*
+ * Routines specialized for GIN case
+ */
+
+static void
+removeabbrev_index_gin(Tuplesortstate *state, SortTuple *stups, int count)
+{
+ Assert(false);
+ elog(ERROR, "removeabbrev_index_gin not implemented");
+}
+
+static int
+comparetup_index_gin(const SortTuple *a, const SortTuple *b,
+ Tuplesortstate *state)
+{
+ Assert(!TuplesortstateGetPublic(state)->haveDatum1);
+
+ return _gin_compare_tuples((GinTuple *) a->tuple,
+ (GinTuple *) b->tuple);
+}
+
+static void
+writetup_index_gin(Tuplesortstate *state, LogicalTape *tape, SortTuple *stup)
+{
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ GinTuple *tuple = (GinTuple *) stup->tuple;
+ unsigned int tuplen = tuple->tuplen;
+
+ tuplen = tuplen + sizeof(tuplen);
+ LogicalTapeWrite(tape, &tuplen, sizeof(tuplen));
+ LogicalTapeWrite(tape, tuple, tuple->tuplen);
+ if (base->sortopt & TUPLESORT_RANDOMACCESS) /* need trailing length word? */
+ LogicalTapeWrite(tape, &tuplen, sizeof(tuplen));
+}
+
+static void
+readtup_index_gin(Tuplesortstate *state, SortTuple *stup,
+ LogicalTape *tape, unsigned int len)
+{
+ GinTuple *tuple;
+ TuplesortPublic *base = TuplesortstateGetPublic(state);
+ unsigned int tuplen = len - sizeof(unsigned int);
+
+ /*
+ * Allocate space for the GIN sort tuple, which already has the proper
+ * length included in the header.
+ */
+ tuple = (GinTuple *) tuplesort_readtup_alloc(state, tuplen);
+
+ tuple->tuplen = tuplen;
+
+ LogicalTapeReadExact(tape, tuple, tuplen);
+ if (base->sortopt & TUPLESORT_RANDOMACCESS) /* need trailing length word? */
+ LogicalTapeReadExact(tape, &tuplen, sizeof(tuplen));
+ stup->tuple = (void *) tuple;
+
+ /* no abbreviations (FIXME maybe use attrnum for this?) */
+ stup->datum1 = (Datum) 0;
+}
+
+
/*
* Routines specialized for DatumTuple case
*/
diff --git a/src/include/access/gin.h b/src/include/access/gin.h
index 25983b7a505..be76d8446f4 100644
--- a/src/include/access/gin.h
+++ b/src/include/access/gin.h
@@ -12,6 +12,8 @@
#include "access/xlogreader.h"
#include "lib/stringinfo.h"
+#include "nodes/execnodes.h"
+#include "storage/shm_toc.h"
#include "storage/block.h"
#include "utils/relcache.h"
@@ -88,4 +90,6 @@ extern void ginGetStats(Relation index, GinStatsData *stats);
extern void ginUpdateStats(Relation index, const GinStatsData *stats,
bool is_build);
+extern void _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc);
+
#endif /* GIN_H */
diff --git a/src/include/access/gin_tuple.h b/src/include/access/gin_tuple.h
new file mode 100644
index 00000000000..56aed40fb96
--- /dev/null
+++ b/src/include/access/gin_tuple.h
@@ -0,0 +1,29 @@
+/*--------------------------------------------------------------------------
+ * gin.h
+ * Public header file for Generalized Inverted Index access method.
+ *
+ * Copyright (c) 2006-2024, PostgreSQL Global Development Group
+ *
+ * src/include/access/gin.h
+ *--------------------------------------------------------------------------
+ */
+#ifndef GIN_TUPLE_
+#define GIN_TUPLE_
+
+#include "storage/itemptr.h"
+
+typedef struct GinTuple
+{
+ Size tuplen; /* length of the whole tuple */
+ Size keylen; /* bytes in data for key value */
+ int16 typlen; /* typlen for key */
+ bool typbyval; /* typbyval for key */
+ OffsetNumber attrnum;
+ signed char category; /* category: normal or NULL? */
+ int nitems; /* number of TIDs in the data */
+ char data[FLEXIBLE_ARRAY_MEMBER];
+} GinTuple;
+
+extern int _gin_compare_tuples(GinTuple *a, GinTuple *b);
+
+#endif /* GIN_TUPLE_H */
diff --git a/src/include/utils/tuplesort.h b/src/include/utils/tuplesort.h
index e7941a1f09f..35fa5ae2442 100644
--- a/src/include/utils/tuplesort.h
+++ b/src/include/utils/tuplesort.h
@@ -22,6 +22,7 @@
#define TUPLESORT_H
#include "access/brin_tuple.h"
+#include "access/gin_tuple.h"
#include "access/itup.h"
#include "executor/tuptable.h"
#include "storage/dsm.h"
@@ -443,6 +444,8 @@ extern Tuplesortstate *tuplesort_begin_index_gist(Relation heapRel,
int sortopt);
extern Tuplesortstate *tuplesort_begin_index_brin(int workMem, SortCoordinate coordinate,
int sortopt);
+extern Tuplesortstate *tuplesort_begin_index_gin(int workMem, SortCoordinate coordinate,
+ int sortopt);
extern Tuplesortstate *tuplesort_begin_datum(Oid datumType,
Oid sortOperator, Oid sortCollation,
bool nullsFirstFlag,
@@ -456,6 +459,7 @@ extern void tuplesort_putindextuplevalues(Tuplesortstate *state,
Relation rel, ItemPointer self,
const Datum *values, const bool *isnull);
extern void tuplesort_putbrintuple(Tuplesortstate *state, BrinTuple *tup, Size len);
+extern void tuplesort_putgintuple(Tuplesortstate *state, GinTuple *tup, Size len);
extern void tuplesort_putdatum(Tuplesortstate *state, Datum val,
bool isNull);
@@ -465,6 +469,8 @@ extern HeapTuple tuplesort_getheaptuple(Tuplesortstate *state, bool forward);
extern IndexTuple tuplesort_getindextuple(Tuplesortstate *state, bool forward);
extern BrinTuple *tuplesort_getbrintuple(Tuplesortstate *state, Size *len,
bool forward);
+extern GinTuple *tuplesort_getgintuple(Tuplesortstate *state, Size *len,
+ bool forward);
extern bool tuplesort_getdatum(Tuplesortstate *state, bool forward, bool copy,
Datum *val, bool *isNull, Datum *abbrev);
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index eee989bba17..9769f4d6b09 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1004,11 +1004,13 @@ GinBtreeData
GinBtreeDataLeafInsertData
GinBtreeEntryInsertData
GinBtreeStack
+GinBuffer
GinBuildState
GinChkVal
GinEntries
GinEntryAccumulator
GinIndexStat
+GinLeader
GinMetaPageData
GinNullCategory
GinOptions
@@ -1021,9 +1023,11 @@ GinScanEntry
GinScanKey
GinScanOpaque
GinScanOpaqueData
+GinShared
GinState
GinStatsData
GinTernaryValue
+GinTuple
GinTupleCollector
GinVacuumState
GistBuildMode
--
2.44.0
[text/x-patch] v20240505-0002-Use-mergesort-in-the-leader-process.patch (9.6K, ../../[email protected]/3-v20240505-0002-Use-mergesort-in-the-leader-process.patch)
download | inline diff:
From f43aeb97f766b24092c3758fa5d6a9f0e6676eaf Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:32 +0200
Subject: [PATCH v20240505 2/8] Use mergesort in the leader process
The leader process (executing the serial part of the index build) spent
a significant part of the time in pg_qsort, after combining the partial
results from the workers. But we can improve this and move some of the
costs to the parallel part in workers - if workers produce sorted TID
lists, the leader can combine them by mergesort.
But to make this really efficient, the mergesort must not be executed
too many times. The workers may easily produce very short TID lists, if
there are many different keys, hitting the memory limit often. So this
adds an intermediate tuplesort pass into each worker, to combine TIDs
for each key and only then write the result into the shared tuplestore.
This means the number of mergesort invocations for each key should be
about the same as the number of workers. We can't really do better, and
it's low enough to keep the mergesort approach efficient.
Note: If we introduce a memory limit on GinBuffer (to not accumulate too
many TIDs in memory), we could end up with more chunks, but it should
not be very common.
---
src/backend/access/gin/gininsert.c | 171 +++++++++++++++++++++++++----
1 file changed, 148 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b353e155fc6..cf7a6278914 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -161,6 +161,14 @@ typedef struct
* build callback etc.
*/
Tuplesortstate *bs_sortstate;
+
+ /*
+ * The sortstate is used only within a worker for the first merge pass
+ * that happens in the worker. In principle it doesn't need to be part of
+ * the build state and we could pass it around directly, but it's more
+ * convenient this way.
+ */
+ Tuplesortstate *bs_worker_sort;
} GinBuildState;
@@ -533,7 +541,7 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
key, attr->attlen, attr->attbyval,
list, nlist, &tuplen);
- tuplesort_putgintuple(buildstate->bs_sortstate, tup, tuplen);
+ tuplesort_putgintuple(buildstate->bs_worker_sort, tup, tuplen);
pfree(tup);
}
@@ -1127,7 +1135,6 @@ typedef struct GinBuffer
/* array of TID values */
int nitems;
- int maxitems;
ItemPointerData *items;
} GinBuffer;
@@ -1136,7 +1143,6 @@ static void
AssertCheckGinBuffer(GinBuffer *buffer)
{
#ifdef USE_ASSERT_CHECKING
- Assert(buffer->nitems <= buffer->maxitems);
#endif
}
@@ -1240,28 +1246,22 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->key = (Datum) 0;
}
- /* enlarge the TID buffer, if needed */
- if (buffer->nitems + tup->nitems > buffer->maxitems)
+ /* copy the new TIDs into the buffer, combine using merge-sort */
{
- /* 64 seems like a good init value */
- buffer->maxitems = Max(buffer->maxitems, 64);
+ int nnew;
+ ItemPointer new;
- while (buffer->nitems + tup->nitems > buffer->maxitems)
- buffer->maxitems *= 2;
+ new = ginMergeItemPointers(buffer->items, buffer->nitems,
+ items, tup->nitems, &nnew);
- if (buffer->items == NULL)
- buffer->items = palloc(buffer->maxitems * sizeof(ItemPointerData));
- else
- buffer->items = repalloc(buffer->items,
- buffer->maxitems * sizeof(ItemPointerData));
- }
+ Assert(nnew == buffer->nitems + tup->nitems);
- /* now we should be guaranteed to have enough space for all the TIDs */
- Assert(buffer->nitems + tup->nitems <= buffer->maxitems);
+ if (buffer->items)
+ pfree(buffer->items);
- /* copy the new TIDs into the buffer */
- memcpy(&buffer->items[buffer->nitems], items, sizeof(ItemPointerData) * tup->nitems);
- buffer->nitems += tup->nitems;
+ buffer->items = new;
+ buffer->nitems = nnew;
+ }
AssertCheckItemPointers(buffer->items, buffer->nitems, false);
}
@@ -1302,6 +1302,21 @@ GinBufferReset(GinBuffer *buffer)
/* XXX should do something with extremely large array of items? */
}
+/* XXX probably would be better to have a memory context for the buffer */
+static void
+GinBufferFree(GinBuffer *buffer)
+{
+ if (buffer->items)
+ pfree(buffer->items);
+
+ /* release byref values, do nothing for by-val ones */
+ if (!GinBufferIsEmpty(buffer) &&
+ (buffer->category == GIN_CAT_NORM_KEY) && !buffer->typbyval)
+ pfree(DatumGetPointer(buffer->key));
+
+ pfree(buffer);
+}
+
/*
* XXX Maybe check size of the TID arrays, and return false if it's too
* large (more thant maintenance_work_mem or something?).
@@ -1375,7 +1390,7 @@ _gin_parallel_merge(GinBuildState *state)
* the data into the insert, and start a new entry for current
* GinTuple.
*/
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer->items, buffer->nitems, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1392,7 +1407,7 @@ _gin_parallel_merge(GinBuildState *state)
/* flush data remaining in the buffer (for the last key) */
if (!GinBufferIsEmpty(buffer))
{
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer->items, buffer->nitems, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1402,6 +1417,9 @@ _gin_parallel_merge(GinBuildState *state)
GinBufferReset(buffer);
}
+ /* relase all the memory */
+ GinBufferFree(buffer);
+
tuplesort_end(state->bs_sortstate);
return reltuples;
@@ -1440,6 +1458,102 @@ _gin_leader_participate_as_worker(GinBuildState *buildstate, Relation heap, Rela
ginleader->sharedsort, heap, index, sortmem, true);
}
+/*
+ * _gin_process_worker_data
+ * First phase of the key merging, happening in the worker.
+ *
+ * Depending on the number of distinct keys, the TID lists produced by the
+ * callback may be very short. But combining many tiny lists is expensive,
+ * so we try to do as much as possible in the workers and only then pass the
+ * results to the leader.
+ *
+ * We read the tuples sorted by the key, and merge them into larger lists.
+ * At the moment there's no memory limit, so this will just produce one
+ * huge (sorted) list per key in each worker. Which means the leader will
+ * do a very limited number of mergesorts, which is good.
+ */
+static void
+_gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
+{
+ GinTuple *tup;
+ Size tuplen;
+
+ GinBuffer *buffer;
+
+ /* initialize buffer to combine entries for the same key */
+ buffer = GinBufferInit();
+
+ /* sort the raw per-worker data */
+ tuplesort_performsort(state->bs_worker_sort);
+
+ /*
+ * Read the GIN tuples from the shared tuplesort, sorted by the key, and
+ * merge them into larger chunks for the leader to combine.
+ */
+ while ((tup = tuplesort_getgintuple(worker_sort, &tuplen, true)) != NULL)
+ {
+
+ CHECK_FOR_INTERRUPTS();
+
+ /*
+ * If the buffer can accept the new GIN tuple, just store it there and
+ * we're done. If it's a different key (or maybe too much data) flush
+ * the current contents into the index first.
+ */
+ if (!GinBufferCanAddKey(buffer, tup))
+ {
+ GinTuple *ntup;
+ Size ntuplen;
+
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ GinBufferSortItems(buffer);
+
+ ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ buffer->key, buffer->typlen, buffer->typbyval,
+ buffer->items, buffer->nitems, &ntuplen);
+
+ tuplesort_putgintuple(state->bs_sortstate, ntup, ntuplen);
+
+ pfree(ntup);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ /* now remember the new key */
+ GinBufferStoreTuple(buffer, tup);
+ }
+
+ /* flush data remaining in the buffer (for the last key) */
+ if (!GinBufferIsEmpty(buffer))
+ {
+ GinTuple *ntup;
+ Size ntuplen;
+
+ GinBufferSortItems(buffer);
+
+ ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ buffer->key, buffer->typlen, buffer->typbyval,
+ buffer->items, buffer->nitems, &ntuplen);
+
+ tuplesort_putgintuple(state->bs_sortstate, ntup, ntuplen);
+
+ pfree(ntup);
+
+ /* discard the existing data */
+ GinBufferReset(buffer);
+ }
+
+ /* relase all the memory */
+ GinBufferFree(buffer);
+
+ tuplesort_end(worker_sort);
+}
+
/*
* Perform a worker's portion of a parallel sort.
*
@@ -1471,6 +1585,10 @@ _gin_parallel_scan_and_build(GinBuildState *state,
state->bs_sortstate = tuplesort_begin_index_gin(sortmem, coordinate,
TUPLESORT_NONE);
+ /* Local per-worker sort of raw-data */
+ state->bs_worker_sort = tuplesort_begin_index_gin(sortmem, NULL,
+ TUPLESORT_NONE);
+
/* Join parallel scan */
indexInfo = BuildIndexInfo(index);
indexInfo->ii_Concurrent = ginshared->isconcurrent;
@@ -1508,7 +1626,7 @@ _gin_parallel_scan_and_build(GinBuildState *state,
key, attr->attlen, attr->attbyval,
list, nlist, &len);
- tuplesort_putgintuple(state->bs_sortstate, tup, len);
+ tuplesort_putgintuple(state->bs_worker_sort, tup, len);
pfree(tup);
}
@@ -1517,6 +1635,13 @@ _gin_parallel_scan_and_build(GinBuildState *state,
ginInitBA(&state->accum);
}
+ /*
+ * Do the first phase of in-worker processing - sort the data produced by
+ * the callback, and combine them into much larger chunks and place that
+ * into the shared tuplestore for leader to process.
+ */
+ _gin_process_worker_data(state, state->bs_worker_sort);
+
/* sort the GIN tuples built by this worker */
tuplesort_performsort(state->bs_sortstate);
--
2.44.0
[text/x-patch] v20240505-0003-Remove-the-explicit-pg_qsort-in-workers.patch (10.9K, ../../[email protected]/4-v20240505-0003-Remove-the-explicit-pg_qsort-in-workers.patch)
download | inline diff:
From 45e7f09ec81932c54eef891017d2a10818dd25b6 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:36 +0200
Subject: [PATCH v20240505 3/8] Remove the explicit pg_qsort in workers
We don't need to do the explicit sort before building the GIN tuple,
because the mergesort in GinBufferStoreTuple is already maintaining the
correct order (this was added in an earlier commit).
The comment also adds a field with the first TID, and modifies the
comparator to sort by it (for each key value). This helps workers to
build non-overlapping TID lists and simply append values instead of
having to do the actual mergesort to combine them. This is best-effort,
i.e. it's not guaranteed to eliminate the mergesort - in particular,
parallel scans are synchronized, and thus may start somewhere in the
middle of the table, and wrap around. In which case there may be very
wide list (with low/high TID values).
Note: There's an XXX comment with a couple ideas on how to improve this,
at the cost of more complexity.
---
src/backend/access/gin/gininsert.c | 124 +++++++++++++++++++++--------
src/include/access/gin_tuple.h | 8 ++
2 files changed, 98 insertions(+), 34 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index cf7a6278914..b2b44066329 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -1110,12 +1110,6 @@ _gin_parallel_heapscan(GinBuildState *state)
return state->bs_reltuples;
}
-static int
-tid_cmp(const void *a, const void *b)
-{
- return ItemPointerCompare((ItemPointer) a, (ItemPointer) b);
-}
-
/*
* State used to combine accumulate TIDs from multiple GinTuples for the same
* key value.
@@ -1147,17 +1141,21 @@ AssertCheckGinBuffer(GinBuffer *buffer)
}
static void
-AssertCheckItemPointers(ItemPointerData *items, int nitems, bool sorted)
+AssertCheckItemPointers(GinBuffer *buffer, bool sorted)
{
#ifdef USE_ASSERT_CHECKING
- for (int i = 0; i < nitems; i++)
+ /* we should not have a buffer with no TIDs to sort */
+ Assert(buffer->items != NULL);
+ Assert(buffer->nitems > 0);
+
+ for (int i = 0; i < buffer->nitems; i++)
{
- Assert(ItemPointerIsValid(&items[i]));
+ Assert(ItemPointerIsValid(&buffer->items[i]));
if ((i == 0) || !sorted)
continue;
- Assert(ItemPointerCompare(&items[i - 1], &items[i]) < 0);
+ Assert(ItemPointerCompare(&buffer->items[i - 1], &buffer->items[i]) < 0);
}
#endif
}
@@ -1220,6 +1218,45 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
return (memcmp(tup->data, DatumGetPointer(buffer->key), buffer->keylen) == 0);
}
+/*
+ * GinBufferStoreTuple
+ * Add data from a GinTuple into the GinBuffer.
+ *
+ * If the buffer is empty, we simply initialize it with data from the tuple.
+ * Otherwise data from the tuple (the TID list) is added to the TID data in
+ * the buffer, either by simply appending the TIDs or doing merge sort.
+ *
+ * The data (for the same key) is expected to be processed sorted by first
+ * TID. But this does not guarantee the lists do not overlap, especially in
+ * the leader, because the workers process interleaving data. But even in
+ * a single worker, lists can overlap - parallel scans require sync-scans,
+ * and if the scan starts somewhere in the table and then wraps around, it
+ * may contain very wide lists (in terms of TID range).
+ *
+ * But the ginMergeItemPointers() is already smart about detecting cases
+ * where it can simply concatenate the lists, and when full mergesort is
+ * needed. And does the right thing.
+ *
+ * By keeping the first TID in the GinTuple and sorting by that, we make
+ * it more likely the lists won't overlap very often.
+ *
+ * XXX How frequent can the overlaps be? If the scan does not wrap around,
+ * there should be no overlapping lists, and thus no mergesort. After ab
+ * overlap, there probably can be many - the one list will be very wide,
+ * with a very low and high TID, and all other lists will overlap with it.
+ * I'm not sure how much we can do to prevent that, short of disabling sync
+ * scans (which for parallel scans is currently not possible). One option
+ * would be to keep two lists of TIDs, and see if the new list can be
+ * concatenated with one of them. The idea is that there's only one wide
+ * list (because the wraparound happens only once), and then do the
+ * mergesort only once at the very end.
+ *
+ * XXX Alternatively, we could simply detect the case when the lists can't
+ * be appended, and flush the current list out. The wrap around happens only
+ * once, so there's going to be only such wide list, and it'll be sorted
+ * first (because it has the lowest TID for the key). So we'd do this at
+ * most once per key.
+ */
static void
GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
{
@@ -1246,7 +1283,12 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->key = (Datum) 0;
}
- /* copy the new TIDs into the buffer, combine using merge-sort */
+ /*
+ * Copy the new TIDs into the buffer, combine with existing data (if any)
+ * using merge-sort. The mergesort is already smart about cases where it
+ * can simply concatenate the two lists, and when it actually needs to
+ * merge the data in an expensive way.
+ */
{
int nnew;
ItemPointer new;
@@ -1261,21 +1303,9 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->items = new;
buffer->nitems = nnew;
- }
-
- AssertCheckItemPointers(buffer->items, buffer->nitems, false);
-}
-static void
-GinBufferSortItems(GinBuffer *buffer)
-{
- /* we should not have a buffer with no TIDs to sort */
- Assert(buffer->items != NULL);
- Assert(buffer->nitems > 0);
-
- pg_qsort(buffer->items, buffer->nitems, sizeof(ItemPointerData), tid_cmp);
-
- AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+ AssertCheckItemPointers(buffer, true);
+ }
}
/* XXX probably would be better to have a memory context for the buffer */
@@ -1299,6 +1329,11 @@ GinBufferReset(GinBuffer *buffer)
buffer->typlen = 0;
buffer->typbyval = 0;
+ if (buffer->items)
+ {
+ pfree(buffer->items);
+ buffer->items = NULL;
+ }
/* XXX should do something with extremely large array of items? */
}
@@ -1390,7 +1425,7 @@ _gin_parallel_merge(GinBuildState *state)
* the data into the insert, and start a new entry for current
* GinTuple.
*/
- AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+ AssertCheckItemPointers(buffer, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1400,14 +1435,17 @@ _gin_parallel_merge(GinBuildState *state)
GinBufferReset(buffer);
}
- /* now remember the new key */
+ /*
+ * Remember data for the current tuple (either remember the new key,
+ * or append if to the existing data).
+ */
GinBufferStoreTuple(buffer, tup);
}
/* flush data remaining in the buffer (for the last key) */
if (!GinBufferIsEmpty(buffer))
{
- AssertCheckItemPointers(buffer->items, buffer->nitems, true);
+ AssertCheckItemPointers(buffer, true);
ginEntryInsert(&state->ginstate,
buffer->attnum, buffer->key, buffer->category,
@@ -1510,7 +1548,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
* the data into the insert, and start a new entry for current
* GinTuple.
*/
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer, true);
ntup = _gin_build_tuple(buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
@@ -1524,7 +1562,10 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinBufferReset(buffer);
}
- /* now remember the new key */
+ /*
+ * Remember data for the current tuple (either remember the new key,
+ * or append if to the existing data).
+ */
GinBufferStoreTuple(buffer, tup);
}
@@ -1534,7 +1575,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinTuple *ntup;
Size ntuplen;
- GinBufferSortItems(buffer);
+ AssertCheckItemPointers(buffer, true);
ntup = _gin_build_tuple(buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
@@ -1835,6 +1876,7 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
tuple->category = category;
tuple->keylen = keylen;
tuple->nitems = nitems;
+ tuple->first = items[0];
/* key type info */
tuple->typlen = typlen;
@@ -1919,6 +1961,12 @@ _gin_parse_tuple(GinTuple *a, ItemPointerData **items)
* assumption that if we get two keys that are two different representations
* of a logically equal value, it'll get merged by the index build.
*
+ * If the key value matches, we compare the first TID value in the TID list,
+ * which means the tuples are merged in an order in which they are most
+ * likely to be simply concatenated. (This "first" TID will also allow us
+ * to determine a point up to which the list is fully determined and can be
+ * written into the index to enforce a memory limit etc.)
+ *
* FIXME Is the assumption we can just memcmp() actually valid? Won't this
* trigger the "could not split GIN page; all old items didn't fit" error
* when trying to update the TID list?
@@ -1953,19 +2001,27 @@ _gin_compare_tuples(GinTuple *a, GinTuple *b)
*/
if (a->typbyval)
{
- return memcmp(&keya, &keyb, a->keylen);
+ int r = memcmp(&keya, &keyb, a->keylen);
+
+ /* if the key is the same, consider the first TID in the array */
+ return (r != 0) ? r : ItemPointerCompare(&a->first, &b->first);
}
else
{
+ int r;
+
if (a->keylen < b->keylen)
return -1;
if (a->keylen > b->keylen)
return 1;
- return memcmp(DatumGetPointer(keya), DatumGetPointer(keyb), a->keylen);
+ r = memcmp(DatumGetPointer(keya), DatumGetPointer(keyb), a->keylen);
+
+ /* if the key is the same, consider the first TID in the array */
+ return (r != 0) ? r : ItemPointerCompare(&a->first, &b->first);
}
}
- return 0;
+ return ItemPointerCompare(&a->first, &b->first);
}
diff --git a/src/include/access/gin_tuple.h b/src/include/access/gin_tuple.h
index 56aed40fb96..8efa33a8d31 100644
--- a/src/include/access/gin_tuple.h
+++ b/src/include/access/gin_tuple.h
@@ -12,6 +12,13 @@
#include "storage/itemptr.h"
+/*
+ * Each worker sees tuples in CTID order, so if we track the first TID and
+ * compare that when combining results in the worker, we would not need to
+ * do an expensive sort in workers (the mergesort is already smart about
+ * detecting this and just concatenating the lists). We'd still need the
+ * full mergesort in the leader, but that's much cheaper.
+ */
typedef struct GinTuple
{
Size tuplen; /* length of the whole tuple */
@@ -20,6 +27,7 @@ typedef struct GinTuple
bool typbyval; /* typbyval for key */
OffsetNumber attrnum;
signed char category; /* category: normal or NULL? */
+ ItemPointerData first; /* first TID in the array */
int nitems; /* number of TIDs in the data */
char data[FLEXIBLE_ARRAY_MEMBER];
} GinTuple;
--
2.44.0
[text/x-patch] v20240505-0004-Compress-TID-lists-before-writing-tuples-t.patch (8.0K, ../../[email protected]/5-v20240505-0004-Compress-TID-lists-before-writing-tuples-t.patch)
download | inline diff:
From 08f8ecd7a21370cc452a6185781428729ad58330 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:39 +0200
Subject: [PATCH v20240505 4/8] Compress TID lists before writing tuples to
disk
When serializing GIN tuples to tuplesorts, we can significantly reduce
the amount of data by compressing the TID lists. The GIN opclasses may
produce a lot of data (depending on how many keys are extracted from
each row), and the compression is very effective and efficient.
If the number of different keys is high, the first worker pass may not
benefit from the compression very much - the data will be spilled to
disk before the TID lists can grow long enough for the compression to
actualy help. In the second pass the impact is much more significant.
For real-world data (full-text on mailing list archives), I usually see
the compression to save only about ~15% in the first pass, but ~50% on
the second pass.
---
src/backend/access/gin/gininsert.c | 116 +++++++++++++++++++++++------
src/tools/pgindent/typedefs.list | 1 +
2 files changed, 95 insertions(+), 22 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b2b44066329..b84fb8f12b6 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -187,7 +187,9 @@ static void _gin_parallel_scan_and_build(GinBuildState *buildstate,
Relation heap, Relation index,
int sortmem, bool progress);
-static Datum _gin_parse_tuple(GinTuple *a, ItemPointerData **items);
+static ItemPointer _gin_parse_tuple_items(GinTuple *a);
+static Datum _gin_parse_tuple_key(GinTuple *a);
+
static GinTuple *_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
Datum key, int16 typlen, bool typbyval,
ItemPointerData *items, uint32 nitems,
@@ -1265,7 +1267,8 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
AssertCheckGinBuffer(buffer);
- key = _gin_parse_tuple(tup, &items);
+ key = _gin_parse_tuple_key(tup);
+ items = _gin_parse_tuple_items(tup);
/* if the buffer is empty, set the fields (and copy the key) */
if (GinBufferIsEmpty(buffer))
@@ -1306,6 +1309,9 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
AssertCheckItemPointers(buffer, true);
}
+
+ /* free the decompressed TID list */
+ pfree(items);
}
/* XXX probably would be better to have a memory context for the buffer */
@@ -1806,6 +1812,15 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
table_close(heapRel, heapLockmode);
}
+/*
+ * Used to keep track of compressed TID lists when building a GIN tuple.
+ */
+typedef struct
+{
+ dlist_node node; /* linked list pointers */
+ GinPostingList *seg;
+} GinSegmentInfo;
+
/*
* _gin_build_tuple
* Serialize the state for an index key into a tuple for tuplesort.
@@ -1818,6 +1833,11 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
* like endianess etc. We could make it a little bit smaller, but it's not
* worth it - it's a tiny fraction of the data, and we need to MAXALIGN the
* start of the TID list anyway. So we wouldn't save anything.
+ *
+ * The TID list is serialized as compressed - it's highly compressible, and
+ * we already have ginCompressPostingList for this purpose. The list may be
+ * pretty long, so we compress it into multiple segments and then copy all
+ * of that into the GIN tuple.
*/
static GinTuple *
_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
@@ -1831,6 +1851,11 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
Size tuplen;
int keylen;
+ dlist_mutable_iter iter;
+ dlist_head segments;
+ int ncompressed;
+ Size compresslen;
+
/*
* Calculate how long is the key value. Only keys with GIN_CAT_NORM_KEY
* have actual non-empty key. We include varlena headers and \0 bytes for
@@ -1854,12 +1879,34 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
else
elog(ERROR, "invalid typlen");
+ /* compress the item pointers */
+ ncompressed = 0;
+ compresslen = 0;
+ dlist_init(&segments);
+
+ /* generate compressed segments of TID list chunks */
+ while (ncompressed < nitems)
+ {
+ int cnt;
+ GinSegmentInfo *seginfo = palloc(sizeof(GinSegmentInfo));
+
+ seginfo->seg = ginCompressPostingList(&items[ncompressed],
+ (nitems - ncompressed),
+ UINT16_MAX,
+ &cnt);
+
+ ncompressed += cnt;
+ compresslen += SizeOfGinPostingList(seginfo->seg);
+
+ dlist_push_tail(&segments, &seginfo->node);
+ }
+
/*
* Determine GIN tuple length with all the data included. Be careful about
- * alignment, to allow direct access to item pointers.
+ * alignment, to allow direct access to compressed segments (those require
+ * SHORTALIGN, but we do MAXALING anyway).
*/
- tuplen = MAXALIGN(offsetof(GinTuple, data) + keylen) +
- (sizeof(ItemPointerData) * nitems);
+ tuplen = MAXALIGN(offsetof(GinTuple, data) + keylen) + compresslen;
*len = tuplen;
@@ -1909,37 +1956,40 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
/* finally, copy the TIDs into the array */
ptr = (char *) tuple + MAXALIGN(offsetof(GinTuple, data) + keylen);
- memcpy(ptr, items, sizeof(ItemPointerData) * nitems);
+ /* copy in the compressed data, and free the segments */
+ dlist_foreach_modify(iter, &segments)
+ {
+ GinSegmentInfo *seginfo = dlist_container(GinSegmentInfo, node, iter.cur);
+
+ memcpy(ptr, seginfo->seg, SizeOfGinPostingList(seginfo->seg));
+
+ ptr += SizeOfGinPostingList(seginfo->seg);
+
+ dlist_delete(&seginfo->node);
+
+ pfree(seginfo->seg);
+ pfree(seginfo);
+ }
return tuple;
}
/*
- * _gin_parse_tuple
- * Deserialize the tuple from the tuplestore representation.
+ * _gin_parse_tuple_key
+ * Return a Datum representing the key stored in the tuple.
*
- * Most of the fields are actually directly accessible, the only thing that
+ * Most of the tuple fields are directly accessible, the only thing that
* needs more care is the key and the TID list.
*
* For the key, this returns a regular Datum representing it. It's either the
* actual key value, or a pointer to the beginning of the data array (which is
* where the data was copied by _gin_build_tuple).
- *
- * The pointer to the TID list is returned through 'items' (which is simply
- * a pointer to the data array).
*/
static Datum
-_gin_parse_tuple(GinTuple *a, ItemPointerData **items)
+_gin_parse_tuple_key(GinTuple *a)
{
Datum key;
- if (items)
- {
- char *ptr = (char *) a + MAXALIGN(offsetof(GinTuple, data) + a->keylen);
-
- *items = (ItemPointerData *) ptr;
- }
-
if (a->category != GIN_CAT_NORM_KEY)
return (Datum) 0;
@@ -1952,6 +2002,28 @@ _gin_parse_tuple(GinTuple *a, ItemPointerData **items)
return PointerGetDatum(a->data);
}
+/*
+* _gin_parse_tuple_items
+ * Return a pointer to a palloc'd array of decompressed TID array.
+ */
+static ItemPointer
+_gin_parse_tuple_items(GinTuple *a)
+{
+ int len;
+ char *ptr;
+ int ndecoded;
+ ItemPointer items;
+
+ len = a->tuplen - MAXALIGN(offsetof(GinTuple, data) + a->keylen);
+ ptr = (char *) a + MAXALIGN(offsetof(GinTuple, data) + a->keylen);
+
+ items = ginPostingListDecodeAllSegments((GinPostingList *) ptr, len, &ndecoded);
+
+ Assert(ndecoded == a->nitems);
+
+ return (ItemPointer) items;
+}
+
/*
* _gin_compare_tuples
* Compare GIN tuples, used by tuplesort during parallel index build.
@@ -1992,8 +2064,8 @@ _gin_compare_tuples(GinTuple *a, GinTuple *b)
if ((a->category == GIN_CAT_NORM_KEY) &&
(b->category == GIN_CAT_NORM_KEY))
{
- keya = _gin_parse_tuple(a, NULL);
- keyb = _gin_parse_tuple(b, NULL);
+ keya = _gin_parse_tuple_key(a);
+ keyb = _gin_parse_tuple_key(b);
/*
* works for both byval and byref types with fixed lenght, because for
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9769f4d6b09..6b67756ebb9 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1023,6 +1023,7 @@ GinScanEntry
GinScanKey
GinScanOpaque
GinScanOpaqueData
+GinSegmentInfo
GinShared
GinState
GinStatsData
--
2.44.0
[text/x-patch] v20240505-0005-Collect-and-print-compression-stats.patch (5.1K, ../../[email protected]/6-v20240505-0005-Collect-and-print-compression-stats.patch)
download | inline diff:
From eaefe4ed07054fd43428f06de35496aecbbadb4a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:43 +0200
Subject: [PATCH v20240505 5/8] Collect and print compression stats
Allows evaluating the benefits of compressing the TID lists.
---
src/backend/access/gin/gininsert.c | 36 +++++++++++++++++++++++++-----
src/include/access/gin.h | 2 ++
2 files changed, 32 insertions(+), 6 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index b84fb8f12b6..2206c47dfb1 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -190,7 +190,8 @@ static void _gin_parallel_scan_and_build(GinBuildState *buildstate,
static ItemPointer _gin_parse_tuple_items(GinTuple *a);
static Datum _gin_parse_tuple_key(GinTuple *a);
-static GinTuple *_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+static GinTuple *_gin_build_tuple(GinBuildState *state,
+ OffsetNumber attrnum, unsigned char category,
Datum key, int16 typlen, bool typbyval,
ItemPointerData *items, uint32 nitems,
Size *len);
@@ -539,7 +540,7 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
/* there could be many entries, so be willing to abort here */
CHECK_FOR_INTERRUPTS();
- tup = _gin_build_tuple(attnum, category,
+ tup = _gin_build_tuple(buildstate, attnum, category,
key, attr->attlen, attr->attbyval,
list, nlist, &tuplen);
@@ -1530,6 +1531,15 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
/* sort the raw per-worker data */
tuplesort_performsort(state->bs_worker_sort);
+ /* print some basic info */
+ elog(LOG, "_gin_parallel_scan_and_build raw %zu compressed %zu ratio %.2f%%",
+ state->buildStats.sizeRaw, state->buildStats.sizeCompressed,
+ (100.0 * state->buildStats.sizeCompressed) / state->buildStats.sizeRaw);
+
+ /* reset before the second phase */
+ state->buildStats.sizeCompressed = 0;
+ state->buildStats.sizeRaw = 0;
+
/*
* Read the GIN tuples from the shared tuplesort, sorted by the key, and
* merge them into larger chunks for the leader to combine.
@@ -1556,7 +1566,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
*/
AssertCheckItemPointers(buffer, true);
- ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ ntup = _gin_build_tuple(state, buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
buffer->items, buffer->nitems, &ntuplen);
@@ -1583,7 +1593,7 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
AssertCheckItemPointers(buffer, true);
- ntup = _gin_build_tuple(buffer->attnum, buffer->category,
+ ntup = _gin_build_tuple(state, buffer->attnum, buffer->category,
buffer->key, buffer->typlen, buffer->typbyval,
buffer->items, buffer->nitems, &ntuplen);
@@ -1598,6 +1608,11 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
/* relase all the memory */
GinBufferFree(buffer);
+ /* print some basic info */
+ elog(LOG, "_gin_process_worker_data raw %zu compressed %zu ratio %.2f%%",
+ state->buildStats.sizeRaw, state->buildStats.sizeCompressed,
+ (100.0 * state->buildStats.sizeCompressed) / state->buildStats.sizeRaw);
+
tuplesort_end(worker_sort);
}
@@ -1669,7 +1684,7 @@ _gin_parallel_scan_and_build(GinBuildState *state,
/* there could be many entries, so be willing to abort here */
CHECK_FOR_INTERRUPTS();
- tup = _gin_build_tuple(attnum, category,
+ tup = _gin_build_tuple(state, attnum, category,
key, attr->attlen, attr->attbyval,
list, nlist, &len);
@@ -1763,6 +1778,7 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
/* initialize the GIN build state */
initGinState(&buildstate.ginstate, indexRel);
buildstate.indtuples = 0;
+ /* XXX shouldn't this initialize the other fiedls, like ginbuild()? */
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
/*
@@ -1840,7 +1856,8 @@ typedef struct
* of that into the GIN tuple.
*/
static GinTuple *
-_gin_build_tuple(OffsetNumber attrnum, unsigned char category,
+_gin_build_tuple(GinBuildState *state,
+ OffsetNumber attrnum, unsigned char category,
Datum key, int16 typlen, bool typbyval,
ItemPointerData *items, uint32 nitems,
Size *len)
@@ -1971,6 +1988,13 @@ _gin_build_tuple(OffsetNumber attrnum, unsigned char category,
pfree(seginfo);
}
+ /* how large would the tuple be without compression? */
+ state->buildStats.sizeRaw += MAXALIGN(offsetof(GinTuple, data) + keylen) +
+ nitems * sizeof(ItemPointerData);
+
+ /* compressed size */
+ state->buildStats.sizeCompressed += tuplen;
+
return tuple;
}
diff --git a/src/include/access/gin.h b/src/include/access/gin.h
index be76d8446f4..2b6633d068a 100644
--- a/src/include/access/gin.h
+++ b/src/include/access/gin.h
@@ -49,6 +49,8 @@ typedef struct GinStatsData
BlockNumber nDataPages;
int64 nEntries;
int32 ginVersion;
+ Size sizeRaw;
+ Size sizeCompressed;
} GinStatsData;
/*
--
2.44.0
[text/x-patch] v20240505-0006-Enforce-memory-limit-when-combining-tuples.patch (14.8K, ../../[email protected]/7-v20240505-0006-Enforce-memory-limit-when-combining-tuples.patch)
download | inline diff:
From fb14d8f86276dc08bec2e93c3191832613a6d56a Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:49 +0200
Subject: [PATCH v20240505 6/8] Enforce memory limit when combining tuples
When combinnig intermediate results during parallel GIN index build, we
want to restrict the memory usage. In ginBuildCallbackParallel() this is
done simply by dumping working state into tuplesort after hitting the
memory limit.
This commit introduces memory limit to the following steps, merging the
intermediate results in both worker and leader. The merge only deals
with one key at a time, and the primary risk is the key might have too
many different TIDs. This is not very likely, because the TID array only
needs 6B per item, it's a potential issue.
We can't simply dump the whole current TID list - the index requires the
TID values to be inserted in the correct order, but if the lists overlap
(as they do between workers), the tail of the list may change during the
mergesort. But thanks to sorting GIN tuples by first TID, we can derive
a safe TID horizon - we know no future tuples will have TIDs from before
this value, so it's safe to output this part of the list.
This commit tracks "frozen" part of the the TID list, which is the part
we know won't change after merging additional TID lists. Then if the TID
list grows too large (more than 64kB), we try to trim it - write out the
frozen part of the list, and discard it from the buffer. We only do the
trimming if there's at least 1024 frozen items - we don't want to write
the data into the index in tiny chunks.
The freezing also allows us to skip the frozen part during mergesort.
The frozen part of the list is known to be fully sorted, so we can just
skip it and mergesort only the rest of the data.
Note: These limites (1024 and 64kB) are mostly arbitrary - but seem high
enough to get good efficiency for compression/batching, but low enough
to release memory early and work in small increments.
---
src/backend/access/gin/gininsert.c | 245 +++++++++++++++++++++++++++--
src/include/access/gin.h | 1 +
2 files changed, 237 insertions(+), 9 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index 2206c47dfb1..f4a4b8f00e9 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -1130,8 +1130,12 @@ typedef struct GinBuffer
int16 typlen;
bool typbyval;
+ /* Number of TIDs to collect before attempt to write some out. */
+ int maxitems;
+
/* array of TID values */
int nitems;
+ int nfrozen;
ItemPointerData *items;
} GinBuffer;
@@ -1166,7 +1170,21 @@ AssertCheckItemPointers(GinBuffer *buffer, bool sorted)
static GinBuffer *
GinBufferInit(void)
{
- return palloc0(sizeof(GinBuffer));
+ GinBuffer *buffer = (GinBuffer *) palloc0(sizeof(GinBuffer));
+
+ /*
+ * How many items can we fit into the memory limit? 64kB seems more than
+ * enough and we don't want a limit that's too high. OTOH maybe this
+ * should be tied to maintenance_work_mem or something like that?
+ *
+ * XXX This is not enough to prevent repeated merges after a wraparound,
+ * but it should be enough to make the merges cheap because it quickly
+ * finds reaches the end of the second list and can just memcpy the rest
+ * without walking it item by item.
+ */
+ buffer->maxitems = (64 * 1024L) / sizeof(ItemPointerData);
+
+ return buffer;
}
static bool
@@ -1221,6 +1239,54 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
return (memcmp(tup->data, DatumGetPointer(buffer->key), buffer->keylen) == 0);
}
+/*
+ * GinBufferShouldTrim
+ * Should we trim the list of item pointers?
+ *
+ * By trimming we understand writing out and removing the tuple IDs that
+ * we know can't change by future merges. We can deduce the TID up to which
+ * this is guaranteed from the "first" TID in each GIN tuple, which provides
+ * a "horizon" (for a given key) thanks to the sort.
+ *
+ * We don't want to do this too often - compressing longer TID lists is more
+ * efficient. But we also don't want to accumulate too many TIDs, for two
+ * reasons. First, it consumes memory and we might exceed maintenance_work_mem
+ * (or whatever limit applies), even if that's unlikely because TIDs are very
+ * small so we can fit a lot of them. Second, and more importantly, long TID
+ * lists are an issue if the scan wraps around, because a key may get a very
+ * wide list (with min/max TID for that key), forcing "full" mergesorts for
+ * every list merged into it (instead of the efficient append).
+ *
+ * So we look at two things when deciding if to trim - if the resulting list
+ * (after adding TIDs from the new tuple) would be too long, and if there is
+ * enough TIDs to trim (with values less than "first" TID from the new tuple),
+ * we do the trim. By enough we mean at least 128 TIDs (mostly an arbitrary
+ * number).
+ *
+ * XXX This does help for the wrap around case, because the "wide" TID list
+ * is essentially two ranges - one at the beginning of the table, one at the
+ * end. And all the other ranges (from GIN tuples) come in between, and also
+ * do not overlap. So by trimming up to the range we're about to add, this
+ * guarantees we'll be able to "concatenate" the two lists cheaply.
+ */
+static bool
+GinBufferShouldTrim(GinBuffer *buffer, GinTuple *tup)
+{
+ /* not enough TIDs to trim (1024 is somewhat arbitrary number) */
+ if (buffer->nfrozen < 1024)
+ return false;
+
+ /* We're not to hit the memory limit after adding this tuple. */
+ if ((buffer->nitems + tup->nitems) < buffer->maxitems)
+ return false;
+
+ /*
+ * OK, we have enough frozen TIDs to flush, and we have hit the memory
+ * limit, so it's time to write it out.
+ */
+ return true;
+}
+
/*
* GinBufferStoreTuple
* Add data from a GinTuple into the GinBuffer.
@@ -1259,6 +1325,11 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
* once, so there's going to be only such wide list, and it'll be sorted
* first (because it has the lowest TID for the key). So we'd do this at
* most once per key.
+ *
+ * XXX Maybe we could/should allocate the buffer once and then keep it
+ * without palloc/pfree. That won't help when just calling the mergesort,
+ * as that does palloc internally, but if we detected the append case,
+ * we could do without it. Not sure how much overhead it is, though.
*/
static void
GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
@@ -1287,26 +1358,82 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
buffer->key = (Datum) 0;
}
+ /*
+ * Try freeze TIDs at the beginning of the list, i.e. exclude them from
+ * the mergesort. We can do that with TIDs before the first TID in the new
+ * tuple we're about to add into the buffer.
+ *
+ * We do this incrementally when adding data into the in-memory buffer,
+ * and not later (e.g. when hitting a memory limit), because it allows us
+ * to skip the frozen data during the mergesort, making it cheaper.
+ */
+
+ /*
+ * Check if the last TID in the current list is frozen. This is the case
+ * when merging non-overlapping lists, e.g. in each parallel worker.
+ */
+ if ((buffer->nitems > 0) &&
+ (ItemPointerCompare(&buffer->items[buffer->nitems - 1], &tup->first) == 0))
+ buffer->nfrozen = buffer->nitems;
+
+ /*
+ * Now search the list linearly, to find the last frozen TID. If we found
+ * the whole list is frozen, this just does nothing.
+ *
+ * Start with the first not-yet-frozen tuple, and walk until we find the
+ * first TID that's higher.
+ *
+ * XXX Maybe this should do a binary search if the number of "non-frozen"
+ * items is sufficiently high (enough to make linear search slower than
+ * binsearch).
+ */
+ for (int i = buffer->nfrozen; i < buffer->nitems; i++)
+ {
+ /* Is the TID after the first TID of the new tuple? Can't freeze. */
+ if (ItemPointerCompare(&buffer->items[i], &tup->first) > 0)
+ break;
+
+ buffer->nfrozen++;
+ }
+
/*
* Copy the new TIDs into the buffer, combine with existing data (if any)
* using merge-sort. The mergesort is already smart about cases where it
* can simply concatenate the two lists, and when it actually needs to
* merge the data in an expensive way.
+ *
+ * XXX We could check if (buffer->nitems > buffer->nfrozen) and only do
+ * the mergesort in that case. ginMergeItemPointers does some palloc
+ * internally, and this way we could eliminate that. But let's keep the
+ * code simple for now.
*/
{
int nnew;
ItemPointer new;
- new = ginMergeItemPointers(buffer->items, buffer->nitems,
+ /*
+ * Resize the array - we do this first, because we'll dereference the
+ * first unfrozen TID, which would fail if the array is NULL. We'll
+ * still pass 0 as number of elements in that array though.
+ */
+ if (buffer->items == NULL)
+ buffer->items = palloc((buffer->nitems + tup->nitems) * sizeof(ItemPointerData));
+ else
+ buffer->items = repalloc(buffer->items,
+ (buffer->nitems + tup->nitems) * sizeof(ItemPointerData));
+
+ new = ginMergeItemPointers(&buffer->items[buffer->nfrozen], /* first unfronzen */
+ (buffer->nitems - buffer->nfrozen), /* num of unfrozen */
items, tup->nitems, &nnew);
- Assert(nnew == buffer->nitems + tup->nitems);
+ Assert(nnew == (tup->nitems + (buffer->nitems - buffer->nfrozen)));
+
+ memcpy(&buffer->items[buffer->nfrozen], new,
+ nnew * sizeof(ItemPointerData));
- if (buffer->items)
- pfree(buffer->items);
+ pfree(new);
- buffer->items = new;
- buffer->nitems = nnew;
+ buffer->nitems += tup->nitems;
AssertCheckItemPointers(buffer, true);
}
@@ -1332,6 +1459,7 @@ GinBufferReset(GinBuffer *buffer)
buffer->category = 0;
buffer->keylen = 0;
buffer->nitems = 0;
+ buffer->nfrozen = 0;
buffer->typlen = 0;
buffer->typbyval = 0;
@@ -1344,6 +1472,23 @@ GinBufferReset(GinBuffer *buffer)
/* XXX should do something with extremely large array of items? */
}
+/*
+ * GinBufferTrim
+ * Discard the "frozen" part of the TID list (which should have been
+ * written to disk/index before this call).
+ */
+static void
+GinBufferTrim(GinBuffer *buffer)
+{
+ Assert((buffer->nfrozen > 0) && (buffer->nfrozen <= buffer->nitems));
+
+ memmove(&buffer->items[0], &buffer->items[buffer->nfrozen],
+ sizeof(ItemPointerData) * (buffer->nitems - buffer->nfrozen));
+
+ buffer->nitems -= buffer->nfrozen;
+ buffer->nfrozen = 0;
+}
+
/* XXX probably would be better to have a memory context for the buffer */
static void
GinBufferFree(GinBuffer *buffer)
@@ -1402,7 +1547,12 @@ _gin_parallel_merge(GinBuildState *state)
/* do the actual sort in the leader */
tuplesort_performsort(state->bs_sortstate);
- /* initialize buffer to combine entries for the same key */
+ /*
+ * Initialize buffer to combine entries for the same key.
+ *
+ * The leader is allowed to use the whole maintenance_work_mem buffer to
+ * combine data. The parallel workers already completed.
+ */
buffer = GinBufferInit();
/*
@@ -1442,6 +1592,36 @@ _gin_parallel_merge(GinBuildState *state)
GinBufferReset(buffer);
}
+ /*
+ * We're about to add a GIN tuple to the buffer - check the memory
+ * limit first, and maybe write out some of the data into the index
+ * first, if needed (and possible). We only flush the part of the TID
+ * list that we know won't change, and only if there's enough data for
+ * compression to work well.
+ *
+ * XXX The buffer may also be empty, but in that case we skip this.
+ */
+ if (GinBufferShouldTrim(buffer, tup))
+ {
+ Assert(buffer->nfrozen > 0);
+
+ state->buildStats.nTrims++;
+
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ AssertCheckItemPointers(buffer, true);
+
+ ginEntryInsert(&state->ginstate,
+ buffer->attnum, buffer->key, buffer->category,
+ buffer->items, buffer->nfrozen, &state->buildStats);
+
+ /* truncate the data we've just discarded */
+ GinBufferTrim(buffer);
+ }
+
/*
* Remember data for the current tuple (either remember the new key,
* or append if to the existing data).
@@ -1465,6 +1645,8 @@ _gin_parallel_merge(GinBuildState *state)
/* relase all the memory */
GinBufferFree(buffer);
+ elog(LOG, "_gin_parallel_merge ntrims " INT64_FORMAT, state->buildStats.nTrims);
+
tuplesort_end(state->bs_sortstate);
return reltuples;
@@ -1525,7 +1707,13 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinBuffer *buffer;
- /* initialize buffer to combine entries for the same key */
+ /*
+ * Initialize buffer to combine entries for the same key.
+ *
+ * The workers are limited to the same amount of memory as during the sort
+ * in ginBuildCallbackParallel. But this probably should be the 32MB used
+ * during planning, just like there.
+ */
buffer = GinBufferInit();
/* sort the raw per-worker data */
@@ -1578,6 +1766,43 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
GinBufferReset(buffer);
}
+ /*
+ * We're about to add a GIN tuple to the buffer - check the memory
+ * limit first, and maybe write out some of the data into the index
+ * first, if needed (and possible). We only flush the part of the TID
+ * list that we know won't change, and only if there's enough data for
+ * compression to work well.
+ *
+ * XXX The buffer may also be empty, but in that case we skip this.
+ */
+ if (GinBufferShouldTrim(buffer, tup))
+ {
+ GinTuple *ntup;
+ Size ntuplen;
+
+ Assert(buffer->nfrozen > 0);
+
+ state->buildStats.nTrims++;
+
+ /*
+ * Buffer is not empty and it's storing a different key - flush
+ * the data into the insert, and start a new entry for current
+ * GinTuple.
+ */
+ AssertCheckItemPointers(buffer, true);
+
+ ntup = _gin_build_tuple(state, buffer->attnum, buffer->category,
+ buffer->key, buffer->typlen, buffer->typbyval,
+ buffer->items, buffer->nfrozen, &ntuplen);
+
+ tuplesort_putgintuple(state->bs_sortstate, ntup, ntuplen);
+
+ pfree(ntup);
+
+ /* truncate the data we've just discarded */
+ GinBufferTrim(buffer);
+ }
+
/*
* Remember data for the current tuple (either remember the new key,
* or append if to the existing data).
@@ -1613,6 +1838,8 @@ _gin_process_worker_data(GinBuildState *state, Tuplesortstate *worker_sort)
state->buildStats.sizeRaw, state->buildStats.sizeCompressed,
(100.0 * state->buildStats.sizeCompressed) / state->buildStats.sizeRaw);
+ elog(LOG, "_gin_process_worker_data trims " INT64_FORMAT, state->buildStats.nTrims);
+
tuplesort_end(worker_sort);
}
diff --git a/src/include/access/gin.h b/src/include/access/gin.h
index 2b6633d068a..9381329fac5 100644
--- a/src/include/access/gin.h
+++ b/src/include/access/gin.h
@@ -51,6 +51,7 @@ typedef struct GinStatsData
int32 ginVersion;
Size sizeRaw;
Size sizeCompressed;
+ int64 nTrims;
} GinStatsData;
/*
--
2.44.0
[text/x-patch] v20240505-0007-Detect-wrap-around-in-parallel-callback.patch (5.7K, ../../[email protected]/8-v20240505-0007-Detect-wrap-around-in-parallel-callback.patch)
download | inline diff:
From f8af37eb2278d3a97b148458161c30122cdbf9e1 Mon Sep 17 00:00:00 2001
From: Tomas Vondra <[email protected]>
Date: Thu, 2 May 2024 15:21:55 +0200
Subject: [PATCH v20240505 7/8] Detect wrap around in parallel callback
When sync scan during index build wraps around, that may result in some
keys having very long TID lists, requiring "full" merge sort runs when
combining data in workers. It also causes problems with enforcing memory
limit, because we can't just dump the data - the index build requires
append-only posting lists, and violating may result in errors like
ERROR: could not split GIN page; all old items didn't fit
because after the scan wrap around some of the TIDs may belong to the
beginning of the list, affecting the compression.
But we can deal with this in the callback - if we see the TID to jump
back, that must mean a wraparound happened. In that case we simply dump
all the data accumulated in memory, and start from scratch.
This means there won't be any tuples with very wide TID ranges, instead
there'll be one tuple with a range at the end of the table, and another
tuple at the beginning. And all the lists in the worker will be
non-overlapping, and sort nicely based on first TID.
For the leader, we still need to do the full merge - the lists may
overlap and interleave in various ways. But there should be only very
few of those lists, about one per worker, making it not an issue.
---
src/backend/access/gin/gininsert.c | 89 ++++++++++++++++++------------
1 file changed, 55 insertions(+), 34 deletions(-)
diff --git a/src/backend/access/gin/gininsert.c b/src/backend/access/gin/gininsert.c
index f4a4b8f00e9..7705eddfa70 100644
--- a/src/backend/access/gin/gininsert.c
+++ b/src/backend/access/gin/gininsert.c
@@ -142,6 +142,7 @@ typedef struct
MemoryContext tmpCtx;
MemoryContext funcCtx;
BuildAccumulator accum;
+ ItemPointerData tid;
/* FIXME likely duplicate with indtuples */
double bs_numtuples;
@@ -474,6 +475,47 @@ ginBuildCallback(Relation index, ItemPointer tid, Datum *values,
MemoryContextSwitchTo(oldCtx);
}
+static void
+ginFlushBuildState(GinBuildState *buildstate, Relation index)
+{
+ ItemPointerData *list;
+ Datum key;
+ GinNullCategory category;
+ uint32 nlist;
+ OffsetNumber attnum;
+ TupleDesc tdesc = RelationGetDescr(index);
+
+ ginBeginBAScan(&buildstate->accum);
+ while ((list = ginGetBAEntry(&buildstate->accum,
+ &attnum, &key, &category, &nlist)) != NULL)
+ {
+ /* information about the key */
+ Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
+
+ /* GIN tuple and tuple length */
+ GinTuple *tup;
+ Size tuplen;
+
+ /* there could be many entries, so be willing to abort here */
+ CHECK_FOR_INTERRUPTS();
+
+ tup = _gin_build_tuple(buildstate, attnum, category,
+ key, attr->attlen, attr->attbyval,
+ list, nlist, &tuplen);
+
+ tuplesort_putgintuple(buildstate->bs_worker_sort, tup, tuplen);
+
+ pfree(tup);
+ }
+
+ MemoryContextReset(buildstate->tmpCtx);
+ ginInitBA(&buildstate->accum);
+}
+
+/*
+ * FIXME Another way to deal with the wrap around of sync scans would be to
+ * detect when tid wraps around and just flush the state.
+ */
static void
ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
bool *isnull, bool tupleIsAlive, void *state)
@@ -484,6 +526,16 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
oldCtx = MemoryContextSwitchTo(buildstate->tmpCtx);
+ /* flush contents before wrapping around */
+ if (ItemPointerCompare(tid, &buildstate->tid) < 0)
+ {
+ elog(LOG, "calling ginFlushBuildState");
+ ginFlushBuildState(buildstate, index);
+ }
+
+ /* remember the TID we're about to process */
+ memcpy(&buildstate->tid, tid, sizeof(ItemPointerData));
+
for (i = 0; i < buildstate->ginstate.origTupdesc->natts; i++)
ginHeapTupleBulkInsert(buildstate, (OffsetNumber) (i + 1),
values[i], isnull[i], tid);
@@ -518,40 +570,7 @@ ginBuildCallbackParallel(Relation index, ItemPointer tid, Datum *values,
* XXX probably should use 32MB, not work_mem, as used during planning?
*/
if (buildstate->accum.allocatedMemory >= (Size) work_mem * 1024L)
- {
- ItemPointerData *list;
- Datum key;
- GinNullCategory category;
- uint32 nlist;
- OffsetNumber attnum;
- TupleDesc tdesc = RelationGetDescr(index);
-
- ginBeginBAScan(&buildstate->accum);
- while ((list = ginGetBAEntry(&buildstate->accum,
- &attnum, &key, &category, &nlist)) != NULL)
- {
- /* information about the key */
- Form_pg_attribute attr = TupleDescAttr(tdesc, (attnum - 1));
-
- /* GIN tuple and tuple length */
- GinTuple *tup;
- Size tuplen;
-
- /* there could be many entries, so be willing to abort here */
- CHECK_FOR_INTERRUPTS();
-
- tup = _gin_build_tuple(buildstate, attnum, category,
- key, attr->attlen, attr->attbyval,
- list, nlist, &tuplen);
-
- tuplesort_putgintuple(buildstate->bs_worker_sort, tup, tuplen);
-
- pfree(tup);
- }
-
- MemoryContextReset(buildstate->tmpCtx);
- ginInitBA(&buildstate->accum);
- }
+ ginFlushBuildState(buildstate, index);
MemoryContextSwitchTo(oldCtx);
}
@@ -587,6 +606,7 @@ ginbuild(Relation heap, Relation index, IndexInfo *indexInfo)
buildstate.bs_numtuples = 0;
buildstate.bs_reltuples = 0;
buildstate.bs_leader = NULL;
+ memset(&buildstate.tid, 0, sizeof(ItemPointerData));
/* initialize the meta page */
MetaBuffer = GinNewBuffer(index);
@@ -2007,6 +2027,7 @@ _gin_parallel_build_main(dsm_segment *seg, shm_toc *toc)
buildstate.indtuples = 0;
/* XXX shouldn't this initialize the other fiedls, like ginbuild()? */
memset(&buildstate.buildStats, 0, sizeof(GinStatsData));
+ memset(&buildstate.tid, 0, sizeof(ItemPointerData));
/*
* create a temporary memory context that is used to hold data not yet
--
2.44.0
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Parallel CREATE INDEX for GIN indexes
@ 2024-05-09 10:14 Andy Fan <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 7+ messages in thread
From: Andy Fan @ 2024-05-09 10:14 UTC (permalink / raw)
To: Tomas Vondra <[email protected]>; +Cc: [email protected]
Tomas Vondra <[email protected]> writes:
> 3) v20240502-0003-Remove-the-explicit-pg_qsort-in-workers.patch
>
> In 0002 the workers still do an explicit qsort() on the TID list before
> writing the data into the shared tuplesort. But we can do better - the
> workers can do a merge sort too. To help with this, we add the first TID
> to the tuplesort tuple, and sort by that too - it helps the workers to
> process the data in an order that allows simple concatenation instead of
> the full mergesort.
>
> Note: There's a non-obvious issue due to parallel scans always being
> "sync scans", which may lead to very "wide" TID ranges when the scan
> wraps around. More about that later.
This is really amazing.
> 7) v20240502-0007-Detect-wrap-around-in-parallel-callback.patch
>
> There's one more efficiency problem - the parallel scans are required to
> be synchronized, i.e. the scan may start half-way through the table, and
> then wrap around. Which however means the TID list will have a very wide
> range of TID values, essentially the min and max of for the key.
>
> Without 0006 this would cause frequent failures of the index build, with
> the error I already mentioned:
>
> ERROR: could not split GIN page; all old items didn't fit
I have two questions here and both of them are generall gin index questions
rather than the patch here.
1. What does the "wrap around" mean in the "the scan may start half-way
through the table, and then wrap around". Searching "wrap" in
gin/README gets nothing.
2. I can't understand the below error.
> ERROR: could not split GIN page; all old items didn't fit
When the posting list is too long, we have posting tree strategy. so in
which sistuation we could get this ERROR.
> issue with efficiency - having such a wide TID list forces the mergesort
> to actually walk the lists, because this wide list overlaps with every
> other list produced by the worker.
If we split the blocks among worker 1-block by 1-block, we will have a
serious issue like here. If we can have N-block by N-block, and N-block
is somehow fill the work_mem which makes the dedicated temp file, we
can make things much better, can we?
--
Best Regards
Andy Fan
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Parallel CREATE INDEX for GIN indexes
@ 2024-05-09 14:28 Tomas Vondra <[email protected]>
parent: Tomas Vondra <[email protected]>
1 sibling, 0 replies; 7+ messages in thread
From: Tomas Vondra @ 2024-05-09 14:28 UTC (permalink / raw)
To: Matthias van de Meent <[email protected]>; +Cc: PostgreSQL Hackers <[email protected]>
On 5/2/24 20:22, Tomas Vondra wrote:
>>
>>> For some of the opclasses it can regress (like the jsonb_path_ops). I
>>> don't think that's a major issue. Or more precisely, I'm not surprised
>>> by it. It'd be nice to be able to disable the parallel builds in these
>>> cases somehow, but I haven't thought about that.
>>
>> Do you know why it regresses?
>>
>
> No, but one thing that stands out is that the index is much smaller than
> the other columns/opclasses, and the compression does not save much
> (only about 5% for both phases). So I assume it's the overhead of
> writing writing and reading a bunch of GB of data without really gaining
> much from doing that.
>
I finally got to look into this regression, but I think I must have done
something wrong before because I can't reproduce it. This is the timings
I get now, if I rerun the benchmark:
workers trgm tsvector jsonb jsonb (hash)
-------------------------------------------------------
0 1225 404 104 56
1 772 180 57 60
2 549 143 47 52
3 426 127 43 50
4 364 116 40 48
5 323 111 38 46
6 292 111 37 45
and the speedup, relative to serial build:
workers trgm tsvector jsonb jsonb (hash)
--------------------------------------------------------
1 63% 45% 54% 108%
2 45% 35% 45% 94%
3 35% 31% 41% 89%
4 30% 29% 38% 86%
5 26% 28% 37% 83%
6 24% 28% 35% 81%
So there's a small regression for the jsonb_path_ops opclass, but only
with one worker. After that, it gets a bit faster than serial build.
While not a great speedup, it's far better than the earlier results that
showed maybe 40% regression.
I don't know what I did wrong before - maybe I had a build with an extra
debug info or something like that? No idea why would that affect only
one of the opclasses. But this time I made doubly sure the results are
correct etc.
Anyway, I'm fairly happy with these results. I don't think it's
surprising there are cases where parallel build does not help much.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2024-05-09 14:28 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2020-06-08 18:39 [PATCH v36 4/6] Index skip scan Dmitrii Dolgov <[email protected]>
2024-05-02 15:19 Parallel CREATE INDEX for GIN indexes Tomas Vondra <[email protected]>
2024-05-02 17:12 ` Re: Parallel CREATE INDEX for GIN indexes Matthias van de Meent <[email protected]>
2024-05-02 18:22 ` Re: Parallel CREATE INDEX for GIN indexes Tomas Vondra <[email protected]>
2024-05-05 18:49 ` Re: Parallel CREATE INDEX for GIN indexes Tomas Vondra <[email protected]>
2024-05-09 14:28 ` Re: Parallel CREATE INDEX for GIN indexes Tomas Vondra <[email protected]>
2024-05-09 10:14 ` Re: Parallel CREATE INDEX for GIN indexes Andy Fan <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox